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ACQUISTON  AND 
TECHNOLOGY 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 
3000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3000 


RISK  MANAGEMENT  GUIDE 


Acquisition  reform  has  changed  the  way  the  Department  of  Defense  designs,  devel¬ 
ops,  manufactures,  and  supports  systems.  Our  technical,  business,  and  management 
approach  for  acquiring  and  operating  systems  has,  and  continues  to,  evolve.  For  ex¬ 
ample,  we  no  longer  can  rely  on  military  specifications  and  standards  to  define  and 
control  how  our  developers  design,  build  and  support  our  new  systems.  Today  we  use 
commercial  hardware  and  software,  promote  open  systems  architecture,  and  encourage 
streamlining  processes,  just  to  name  a  few  of  the  initiatives  that  affect  the  way  we  do 
business.  At  the  same  time,  the  Office  of  the  Secretary  of  Defense  has  reduced  the  level 
of  oversight  and  review  of  programs  and  manufacturers'  plants. 

While  the  new  acquisition  model  gives  government  Program  Managers  and  their 
contractors  broader  control  and  more  options  than  they  have  enjoyed  in  the  past,  it  also 
exposes  them  to  new  risks.  OSD  recognizes  that  risk  is  inherent  in  any  acquisition 
program  and  considers  it  essential  that  Program  Managers  take  appropriate  steps  to 
manage  and  control  risks. 

In  late  1996,  the  Under  Secretary  of  Defense,  Acquisition  and  Technology 
[USD(A&T)]  tasked  the  Director,  Test,  Systems  Engineering,  and  Evaluation  (DTSE&E) 
to  review  DoD  risk  management  practices  and  techniques.  In  response,  DTSE&E/ 
Systems  Engineering  established  a  Risk  Management  Working  Group  that  examined 
the  Services,  individual  acquisition  programs,  and  commercial  industry's  treatment  of 
risk.  The  results  of  the  study  served  as  the  basis  for  the  risk  management  section  (2.5.2) 
in  the  Defense  Acquisition  Deskbook.  The  study  also  identified  the  need  to  update 
existing  risk  training  material  to  reflect  the  new  way  DoD  conducts  business. 


This  document  is  a  product  of  a  joint  effort  among  the  DTSE&E,  the  Defense 
Acquisition  University,  and  the  Defense  Systems  Management  College  (DSMC).  It  is 
Based  on  the  material  developed  by  the  DoD  Risk  Management  Working  Group,  in¬ 
cluded  in  the  Defense  Acquisition  Deskbook. 

Mark  D.  Scheffer 
Deputy  Director,  Test,  Systems 
Engineering  and  Evaluation/ 

Systems  Engineering 


RADM,  SC,  USN 
Commandant 

Defense  Systems  Management  College 


President 

Defense  Acquisition  University 
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PREFACE 


In  December  1995,  the  Under  Secretary  of  Defense,  Acquisition  and  Technology  [USD(A&T)], 
issued  a  memorandum  entitled  Reducing  Life  Cycle  Costs  for  New  and  Fielded  Systems,  in  which 
he  established  the  policy  and  strategy  to  develop  and  field  affordable  weapon  systems  that 
are  responsive  to  user's  needs.  One  of  the  foundations  of  the  strategy  is  the  concept  of  "Cost 
as  An  Independent  Variable"  (CAIV),  the  Department  of  Defense  (DoD)  equivalent  of  com¬ 
mercial  best  practices.  The  CAIV  concept  recognizes  that  "there  are  risks  to  be  taken  and  risks 
to  be  avoided.  When  risks  are  taken,  we  will  put  in  place  appropriate  risk  management  and 
contingency  plans." 

Other  initiatives,  such  as  acquisition  streamlining  and  revision  of  the  DoD  5000  series 
documents,  were  ongoing  when  the  USD(A&T)  memorandum  was  published;  each  af¬ 
fected  program  risk.  Also  at  this  time,  the  DoD  Inspector  General  was  writing  a  critical 
report  of  the  Department's  management  of  risk;  the  report  recommended  measures  to 
control  risk  of  acquisition  programs.  Figure  P-1  shows  some  of  the  initiatives  that  impact 
risk  management. 


ACQUISITION  INITIATIVES 

Emphasis  on 

Risk 

OPEN  SYSTEMS 

Management 

COST  AS  AN  INDEPENDENT  VARIABLE  (CAIV) 

1 - 

ACQUISITION  STREAMLINING 

COMMERCIAL  ITEMS,  HARDWARE  AND  SOFTWARE 


MILITARY  SPECIFICATIONS  AND  STANDARDS 
REDUCED  PLANT  OVERSIGHT 


SINGLE  PROCESS  INITIATIVE 


REDUCTION  IN  FUNDING 


Figure  P-1.  DoD  Renewed  Emphasis  on  Risk  Management 

With  these  initiatives  as  the  basis,  the  USD(A&T)  tasked  the  Director,  Test,  Systems  Engi¬ 
neering,  and  Evaluation  (DTSE&E)  to  (1)  review  DoD  risk  management  practices  and 
techniques,  (2)  determine  whether  new  approaches  were  needed  to  improve  risk  man¬ 
agement,  and  (3)  report  the  results  to  USD(A&T). 

In  response,  DTSE&E  established  a  Risk  Management  Working  Group  composed  of  mem¬ 
bers  of  the  OSD  staff,  representatives  of  the  Services,  and  members  of  other  DoD  agencies 


V 


involved  in  systems  acquisition.  This  group  reviewed  pertinent  DoD  directives  (DoDD) 
and  regulations,  examined  how  the  Services  managed  risk,  studied  various  examples  of 
risk  management  by  companies  in  commercial  industry,  and  looked  at  DoD  training  and 
education  activity  in  risk  management.  The  Working  Group  coordinated  with  other  re¬ 
lated  efforts  in  DoD.  For  example,  the  Joint  Aeronautical  Commanders  Group  Risk  Guide 
was  a  valuable  source  of  information.  The  workshops  for  the  CAIV  Flagship  programs 
provided  ciurcnt,  real-world  examples  of  Program  Mangers  implementing  the  CAIV  ini¬ 
tiative  and  risk  management.  Membership  of  the  Working  Group  included  a  representa¬ 
tive  from  USD(A&T)  Acquisition  Program  Integration/Program  Management  (API/PM) 
who  kept  members  informed  on  the  status  of  the  Integrated  Program  Management  Initia¬ 
tive.  Other  sources  of  information  were  the  Software  Engineering  Institute  Risk  Initia¬ 
tive,  the  Open  Systems  Initiative,  and  Safety  and  Cost  Estimating  communities.  DTSE&E 
summarized  the  findings  of  the  investigation  and  presented  the  results  to  the  USD(A&T) 
in  July  1996. 

The  findings  and  recommendations  of  the  Working  Group  are  summarized  below. 


Commercial  IriduistriM 


•  Focus  of  efforts  is  on  getting  a  product  to  market  at  a  competitive  cost. 

•  Companies  have  either  a  structured  or  informal  Risk  Management  process. 

•  Evolutionary  approaches  help  avoid  or  minimize  risk. 

•  Most  approaches  employ  risk  avoidance,  early  planning,  continuous  assessment,  and 
problem-solving  techniques. 

•  Structured  approaches,  when  they  exist,  are  similar  to  DoD’s  approach  to  Risk 
Management. 

The  Working  Group  concluded  that  industry  has  no  magic  formula  for  Risk  Management. 


The  Services 


•  The  Services  differ  in  their  approaches  to  Risk  Management. 

•  Each  approach  has  its  strengths  but  no  one  approach  is  comprehensive. 

•  Consolidation  of  the  strengths  of  each  approach  couid  foster  better  Risk  Management  in 
DoD. 

The  Working  Group  recommended  that  the  Defense  Acquisition  Deskbook  contain  a  set  of 
guidelines  for  sound  risk  management  practices,  and  further,  that  it  contain  a  set  of  risk 
management  definitions  that  are  comprehensive  and  usefui  by  all  the  Components. 


.it: 


PoDPbiiCy 


•  The  risk  management  policy  contained  in  DoDD  5000.1  is  not  comprehensive. 

The  Working  Group  recommended  that  DoDD  5000.1  be  amended  to  inciude  a  more 
comprehensive  set  of  risk  management  poiicies  that  focuses  on: 

•  The  relationship  between  the  CAIV  concept  and  Risk  Management 

•  Requirement  that  risk  management  be  prospective  (fon/vard  looking) 

•  Establishment  of  risk  management  as  a  primary  management  technique  to  be  used  by 

Program  Managers  (PMs). _ 
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OoD  Procedures  '< 


•  Risk  Management  procedures  in  DoD  5000.2-R  are  inadequate  to  fully  implement  the  risk 
management  policy  contained  in  DoDD  5000.1 

•  Procedures  are  lacking  regarding  the: 

-  Scope  of  Risk  Management 

-  Purpose  of  Risk  Management 

-  Role  of  Milestone  Decision  Authorities 

-  Risk  Management’s  support  of  CAIV 

-  Risk  assessment  during  Phase  0. 

•  Some  key  procedures  may  have  been  lost  in  transition  from  DoD  5000.2M  to  DoD 
5000.2-R. 


The  Working  Group  recommended  that  procedures  in  DoD  5000.2-R  be  expanded,  using  the 
Defense  Acquisition  Deskbook  as  the  expansion  means,  in  order  to  provide  comprehensive 
guidance  for  the  implementation  of  risk  management  policy. _ _ 


PoD  Risk  Management  Training 


fc. 


•  Risk  management  training  for  the  DoD  acquisition  corps  needs  to  be  updated  and 
expanded,  and  IPT  and  OIPT  personnel  need  to  be  educated  on  the  new  and  expanding 
role  of  risk  management  in  DoD  systems  acquisition. 

•  Risk  Management  knowledge  level  needs  improvement. 

•  Education  is  a  key  to  getting  the  support  of  OIPTs  and  PMs. 


The  Working  Group  recommended  that  Defense  Acquisition  University  (DAU)  include  training 
for  Risk  Management  In  all  functional  courses  and  develop  a  dedicated  risk  management 
course  for  acquisition  corps  personnel. 


DTSE&E  briefed  the  results  to  the  Defense  Manufacturing  Council,  an  advisory  body  to  the 
USD(A&T),  which  directed  that  the  recommendations  be  incorporated  in  the  Defense  Acqui¬ 
sition  Deskbook.  Following  that  guidance,  DTSE&E  wrote  the  risk  management  portions  of 
the  Deskbook. 

The  Risk  Deskbook  write-up  forms  the  basis  for  this  guide.  The  goal  of  the  Risk  Manage¬ 
ment  Guide  is  to  provide  acquisition  professionals  and  program  management  offices  with 
a  reference  for  dealing  with  system  acquisition  risks.  It  has  been  designed  as  an  aid  in 
classroom  instruction  and  as  a  reference  for  practical  applications. 

This  guide  reflects  the  efforts  of  many  people.  Mr.  Mark  Schaeffer,  Deputy  Director,  Sys¬ 
tems  Engineering,  DTSE&E,  who  chaired  the  Risk  Management  Working  Group  and  Mr. 
Mike  Zsak,  from  the  DTSE&E,  Systems  Engineering  Support  Office,  were  the  driving 
force  behind  the  risk  management  initiative.  Paul  McMahon  and  Bill  Bahnmaier  from  the 
DSMC  faculty  and  DSMC  Press  guided  the  composition  of  the  Guide.  Special  recognition 
goes  to  the  Institute  for  Defense  Analyses  team  composed  of  Louis  Simpleman,  Ken  Evans, 
Jim  Lloyd,  and  Gerald  Pike,  who  compiled  the  data  and  wrote  major  portions  of  the  text. 
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INTRODUCTION 


Risk  has  always  been  a  concern  in  the  ac¬ 
quisition  of  DoD  systems.  The  acquisition 
process  itself  is  designed,  to  a  large  degree, 
to  allow  risks  to  be  controlled  from  con¬ 
ception  to  delivery  of  a  system.  Unfortu¬ 
nately,  in  the  past,  some  Program  Manag¬ 
ers  and  decision  makers  have  viewed  risk 
as  something  to  be  avoided.  Any  program 
that  had  risk  was  subject  to  intense  review 
and  oversight.  This  attitude  has  changed. 
DoD  managers  recognize  that  risk  is  inher¬ 
ent  in  any  program  and  that  it  is  necessary 
to  analyze  future  program  events  to  iden¬ 
tify  potential  risks  and  take  measures  to 
handle  them. 

Risk  management  is  concerned  with  the 
outcome  of  future  events,  whose  exact  out¬ 
come  is  unknown,  and  with  how  to  deal 
with  these  uncertainties,  i.e.,  a  range  of 
possible  outcomes.  In  general,  outcomes 
are  categorized  as  favorable  or  unfavor¬ 
able,  and  risk  management  is  the  art  and 
science  of  planning,  assessing,  and  han¬ 
dling  future  events  to  ensure  favorable  out¬ 
comes.  The  alternative  to  risk  management 
is  crisis  management,  a  resource-intensive 
process  that  is  normally  constrained  by  a 
restricted  set  of  available  options. 

1.1  PURPOSE  AND  SCOPE 

This  Risk  Management  Guide  is  designed  to 
provide  acquisition  professionals  and  pro¬ 
gram  management  offices  with  a  reference 
book  for  dealing  with  system  acquisition 
risks.  It  is  intended  to  be  useful  as  an  aid  in 


classroom  instruction  and  as  a  reference  book 
for  practical  applications.  Most  of  the  mate¬ 
rial  in  this  guide  is  derived  from  the  Defense 
Acquisition  Deskbook.  Readers  should  re¬ 
fer  to  Paragraph  2.5.2  of  the  Deskbook  for 
any  new  information. 

1.2  ORGANIZATION  OF  THE  GUIDE 

The  Risk  Management  Guide  discusses  risk  and 
risk  management,  defines  terms,  and  intro¬ 
duces  basic  risk  management  concepts 
(Chapter  2). 

Chapter  3  examines  risk  management  con¬ 
cepts  relative  to  the  DoD  acquisition  process. 
It  illustrates  how  risk  management  is  an  in¬ 
tegral  part  of  program  management,  de¬ 
scribes  interaction  with  other  acquisition  pro¬ 
cesses,  and  identifies  and  discusses  the  vari¬ 
ous  types  of  acquisition  risks. 

Chapter  4  discusses  the  implementation  of  a 
risk  management  program  from  the  perspec¬ 
tive  of  a  program  management  office  (PMO). 
This  chapter  focuses  on  practical  application 
issues  such  as  risk  management  program 
design  options,  PMO  risk  management  or¬ 
ganizations,  and  criteria  for  a  risk-manage¬ 
ment  information  system  (MIS). 

Chapter  5,  the  final  chapter,  describes  a  num¬ 
ber  of  techniques  that  address  the  aspects  of 
risk  management,  i.e.,  planning,  identifying, 
analyzing,  handling,  and  monitoring. 

This  Guide  is  a  source  of  backgroimd  infor¬ 
mation  and  provides  a  starting  point  for  a 
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risk  management  program.  None  of  the  ma¬ 
terial  is  mandatory.  Program  Managers 
should  tailor  the  approaches  and  techniques 
to  fit  their  programs. 

The  Risk  Management  Guide  also  contains 
appendices  that  are  intended  to  serve  as 
reference  material  and  provide  backup  de¬ 
tail  for  some  of  the  concepts  presented  in 
the  main  portion  of  the  Guide. 

1.3  APPROACH  TO  RISK 
MANAGEMENT 

Based  on  the  DoD  model  contained  in  the 
Defense  Acquisition  Deskbook  (DAD)  (de¬ 
scribed  in  Chapter  2),  this  Guide  emphasizes 
a  risk  management  approach  that  is  disci¬ 
plined,  forward  looking,  and  continuous. 

In  1986,  the  Government  Accounting  Of¬ 
fice  (GAO),  as  part  of  an  evaluation  of  DoD 
policies  and  procedures  for  technical  risk 
assessments,  developed  a  set  of  criteria  as 
an  approach  to  good  risk  assessments. 
These  criteria,  with  slight  modification, 
apply  to  all  aspects  of  risk  management 
and  are  encompassed  in  the  Guide's  ap¬ 
proach.  They  are: 

(1)  Planned  Procedures.  Risk  manage¬ 
ment  is  planned  and  systematic. 

(2)  Prospective  Assessment.  Potential 
future  problems  are  considered,  not  just 
current  problems. 

(3)  Attention  to  Technical  Risk.  There 
is  explicit  attention  to  technical  risk. 

(4)  Documentation.  All  aspects  of  the 
risk  management  program  are  recorded 
and  data  maintained. 

(5)  Continual  Process.  Risk  assess¬ 
ments  are  made  throughout  the  acquisition 


process;  handling  activities  are  continually 
evaluated  and  changed  if  necessary;  and 
critical  risk  areas  are  always  monitored. 

1.4  DOD  RISK  MANAGEMENT 
POLICIES  AND  PROCEDURES 

DoD  policies  and  procedures  that  address 
risk  management  for  acquisition  programs 
are  contained  in  four  key  DoD  documents. 
DoDD  5000.1  contains  the  policy  on  risk 
management  and  is  amplified  further  by 
the  information  in  DoD  5000.2-R.  The  lat¬ 
ter  document  integrates  risk  management 
into  the  acquisition  process,  describes  the 
relationship  between  risk  and  various  ac¬ 
quisition  functions,  and  establishes  some 
reporting  requirements.  DoDD  5000.4  and 
DoD  5000.4-M  address  risk  and  cost  analy¬ 
sis  guidance  as  they  apply  to  the  Office  of 
the  Secretary  of  Defense.  Appendix  A  is 
an  extract  of  existing  risk  management 
policies  and  procedures  from  all  of  these 
documents. 

The  DoD  5000  series  contains  strong  state¬ 
ments  on  risk  management  but  requires 
elaboration  to  help  the  Program  Manager 
(PM)  establish  an  effective  risk  manage¬ 
ment  program.  The  information  furnished 
in  the  Risk  Management  section  of  the 
Deskbook  supports  and  expands  the  con¬ 
tents  of  the  DoD  5000  series. 

The  DoD  risk  management  policies  and 
procedures  provide  the  basis  for  this 
Guide,  which  complements  the  Deskbook 
by  elaborating  on  risk  management  con¬ 
cepts  and  by  providing  greater  detail  for 
applying  techniques. 
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RISK  AND  RISK  MANAGEMENT 


2.1  INTRODUCTION 

This  Chapter  introduces  the  concepts  of 
risk  and  risk  management  by  explaining 
the  DoD  risk-related  definitions  and  by 
identifying  the  characteristics  of  acquisition 
risks.  It  also  presents  and  discusses  a  struc¬ 
tured  concept  for  risk  management  and  its 
five  subordinate  processes. 

2.2  OVERVIEW 

The  DoD  risk  management  concept  is 
based  on  the  principles  that  risk  manage¬ 
ment  must  be  forward-looking,  structured, 
informative,  and  continuous.  The  key  to 
successful  risk  management  is  early  plan¬ 
ning  and  aggressive  execution.  Good  plan¬ 
ning  enables  an  organized,  comprehensive, 
and  iterative  approach  for  identifying  and 
assessing  the  risk  and  handling  options 
necessary  to  refine  a  program  acquisition 
strategy.  To  support  these  efforts,  assess¬ 
ments  should  be  performed  as  early  as 
possible  in  the  life  cycle  to  ensure  that  criti¬ 
cal  technical,  schedule,  and  cost  risks  are 
addressed  with  mitigation  actions  incorpo¬ 
rated  into  program  plaiming  and  budget 
projections. 

PMs  should  update  program  risk  assess¬ 
ments  and  tailor  their  management  strate¬ 
gies  accordingly.  Early  information  gives 
them  data  that  helps  when  writing  a  Re¬ 
quest  for  Proposal  and  assists  in  Source  Se¬ 
lection  planning.  As  a  program  progresses, 
new  information  improves  insight  into  risk 
areas,  thereby  allowing  the  development 


of  effective  handling  strategies.  The  net  re¬ 
sult  promotes  executable  programs. 

Effective  risk  management  requires  in¬ 
volvement  of  the  entire  program  team  and 
also  requires  help  from  outside  experts 
knowledgeable  in  critical  risk  areas  (e.g., 
threat,  technology,  design,  manufacturing, 
logistics,  schedule,  and  cost).  In  addition, 
the  risk  management  process  should  cover 
hardware,  software,  the  human  element, 
and  integration  issues.  Outside  experts 
may  include  representatives  from  the  user, 
laboratories,  contract  management,  test, 
logistics,  and  sustainment  communities, 
and  industry.  Users,  essential  participants 
in  program  trade  analyses,  should  be  part 
of  the  assessment  process  so  that  an  accept¬ 
able  balance  among  performance,  cost, 
schedule,  and  risk  can  be  reached.  A  close 
relationship  between  the  Government  and 
industry,  and  later  with  the  selected 
contractor(s),  promotes  an  understanding 
of  program  risks  and  assists  in  developing 
and  executing  the  management  efforts. 

Successful  risk  management  programs  gen¬ 
erally  have  the  following  characteristics: 

•  Feasible,  stable,  and  well-understood 
user  reqijirements  and  threat; 

•  A  close  relationship  with  user,  indus¬ 
try,  and  other  appropriate  participants; 

•  A  planned  risk  management  process, 
integral  to  the  acquisition  process; 
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•  An  acquisition  strategy  consistent  with 
risk  level  and  risk-handling  strategies; 

•  Continual  reassessment  of  program 
and  associated  risks; 

•  A  defined  set  of  success  criteria  for  all 
performance,  schedule,  and  cost  elements, 
e.g..  Acquisition  Program  Baseline  (APB) 
thresholds; 

•  Metrics  to  monitor  effectiveness  of 
risk-handling  strategies; 

•  Effective  Test  and  Evaluation  Program; 

•  Formal  documentation. 

PMs  should  follow  the  guidelines  below 
to  ensure  that  a  management  program  pos¬ 
sesses  the  above  characteristics. 

•  Assess  program  risks,  using  a  structured 
process,  and  develop  strategies  to  manage 
these  risks  throughout  each  acquisition 
phase. 

•  Identify  early  and  intensively  manage 
those  design  parameters  that  critically  af¬ 
fect  capability,  readiness,  or  cost. 

•  Use  technology  demonstrations/mod¬ 
eling/simulation  and  aggressive  proto-typ¬ 
ing  to  reduce  risks. 

•  Use  test  and  evaluation  as  a  means  of 
quantifying  the  results  of  the  risk-mitiga¬ 
tion  process. 

•  Include  industry  and  user  participa¬ 
tion  in  risk  management. 

•  Use  Developmental  Test  and  Evalua¬ 
tion  (DT&E)  and  early  operational  assess¬ 
ments  when  appropriate. 

•  Establish  a  series  of  "risk  assessment 
reviews"  to  evaluate  the  effectiveness  of 
risk  handling  against  clearly  defined  suc¬ 
cess  criteria. 

•  Establish  the  means  and  format  to  com¬ 
municate  risk  information  and  to  train  par¬ 
ticipants  in  risk  management. 


•  Prepare  an  assessment  training  pack¬ 
age  for  members  of  the  program  office  and 
others,  as  needed. 

•  Acquire  approval  of  accepted  risks  at 
the  appropriate  decision  level. 

In  general,  management  of  software  risk  is 
the  same  as  management  of  other  types  of 
risk  and  techniques  that  apply  to  hardware 
programs  are  equally  applicable  to  soft¬ 
ware  intensive  programs.  However,  some 
characteristics  of  software  make  this  t5q>e 
of  risk  management  different,  primarily 
because  it  is  difficult  to: 

•  Identify  software  risk. 

•  Estimate  the  time  and  resources  re¬ 
quired  to  develop  new  software,  resulting  in 
potential  risks  in  schedule  and  cost. 

•  Test  software  completely  because  of 
the  number  of  paths  that  can  be  followed 
in  the  logic  of  the  software. 

•  Develop  new  programs  because  of  the 
rapid  changes  in  information  technology 
and  an  ever-increasing  demand  for  qual¬ 
ity  software  personnel. 

2.3  RISK  MANAGEMENT 

STRUCTURE  AND  DEFINITIONS 

Although  each  risk  management  strategy 
depends  upon  the  nature  of  the  system 
being  developed,  research  reveals  that 
good  strategies  contain  the  same  basic  pro¬ 
cesses  and  structure  shown  in  Figure  2-1. 
The  application  of  these  processes  vary 
with  acquisition  phases  and  the  degree  of 
system  definition;  all  should  be  integrated 
into  the  program  management  function. 

The  elements  of  the  structure  are  discussed 
in  the  following  paragraphs  of  this  Chapter; 
however,  in  order  to  form  a  basis  for  discus¬ 
sion,  the  Defense  Acquisition  Deskbook  defi¬ 
nitions  for  the  processes  and  elements  of  risk 
management  include: 
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Risk  is  a  measure  of  the  potential  inabil¬ 
ity  to  achieve  overall  program  objectives 
within  defined  cost,  schedule,  and  techni¬ 
cal  constraints  and  has  two  components: 
(1)  the  probability  of  failing  to  achieve  a  par¬ 
ticular  outcome  and  (2)  the  consequences  of 
failing  to  achieve  that  outcome. 

Risk  management  is  the  act  or  practice 
of  dealing  with  risk.  It  includes  planning 
for  risk,  assessing  risk  areas,  developing 
risk-handling  options,  monitoring  risks  to 
determine  how  risks  have  changed,  and 
documenting  the  overall  risk  management 
program. 

Risk  planning  is  the  process  of  develop¬ 
ing  and  documenting  an  organized,  compre¬ 
hensive,  and  interactive  strategy  and  meth¬ 
ods  for  identif3nng  and  tracking  risk  areas, 
developing  risk-mitigation  plans,  perform¬ 
ing  continuous  risk  assessments  to  determine 
how  risks  have  changed,  and  assigning  ad¬ 
equate  resomces. 

Risk  events,  things  that  could  go  wrong 
for  a  program  or  system,  are  elements  of 
an  acquisition  program  that  should  be  as¬ 
sessed  to  determine  the  level  of  risk.  The 
events  should  be  defined  to  a  level  that  an 
individual  can  comprehend  the  potential 
impact  and  its  causes.  For  example,  a  po¬ 
tential  risk  event  for  a  turbine  engine  could 


be  turbine  blade  vibration.  There  could  be 
a  series  of  potential  risk  events  that  should 
be  selected,  examined,  and  assessed  by 
subject  matter  experts. 

The  relationship  between  the  two  compo¬ 
nents  of  risk,  probability  and  consequence, 
is  complex.  To  avoid  obscuring  the  results 
of  an  assessment,  the  risk  associated  with  an 
event  should  be  characterized  in  terms  of  its 
two  components.  There  is  stiU  a  need  for 
backup  documentation  containing  the  sup¬ 
porting  data  and  assessment  rationale. 

Risk  assessment  is  the  process  of  iden¬ 
tifying  and  analyzing  program  areas  and 
critical  technical  process  risks  to  increase 
the  likelihood  of  meeting  performance, 
schedule,  and  cost  objectives.  Risk  identifi¬ 
cation  is  the  process  of  examining  the  pro¬ 
gram  areas  and  each  critical  technical  pro¬ 
cess  to  identify  and  document  the  associ¬ 
ated  risk.  Risk  analysis  is  the  process  of  ex¬ 
amining  each  identified  risk  area  or  pro¬ 
cess  to  refine  the  description  of  the  risk, 
isolating  the  cause,  and  determining  the 
effects.  Risk  priority  is  defined  in  terms  of 
its  probability  of  occurrences,  its  conse¬ 
quences,  and  its  relationship  to  other  risk 
areas  or  processes. 

Risk  handling  is  the  process  that  identi¬ 
fies,  evaluates,  selects,  and  implements  op- 
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tions  in  order  to  set  risk  at  acceptable  levels 
given  program  constraints  and  objectives. 
This  includes  the  specifics  on  what  should 
be  done,  when  it  should  be  accomplished, 
who  is  responsible,  and  associated  cost.  The 
most  appropriate  strategy  is  selected  from 
these  handling  options.  For  purposes  of  the 
Guide,  risk  handling  is  an  aU-encompassing 
term  whereas  risk  mitigation  and  risk  con¬ 
trol  are  subsets  of  risk  handling. 

Risk  monitoring  is  the  process  that  sys¬ 
tematically  tracks  and  evaluates  the  per¬ 
formance  of  risk-handling  actions  against 
established  metrics  throughout  the  acqui¬ 
sition  process  and  develops  further  risk¬ 
handling  options,  as  appropriate. 

Risk  documentation  is  recording,  main¬ 
taining,  and  reporting  assessments,  han¬ 
dling  analysis  and  plans,  and  monitoring 
results.  It  includes  all  plans,  reports  for  the 
Program  Manager  and  decision  authorities, 
and  reporting  forms  that  may  be  internal 
to  the  program  management  office. 

2.4  RISK  DISCUSSION 

Implicit  in  the  definition  of  risk  is  the  con¬ 
cept  that  risks  are  future  events  and  that 
there  is  uncertainty  associated  with  the  pro¬ 
gram  if  these  events  occur.  Therefore,  there 
is  a  need  to  determine,  as  much  as  possible, 
the  probability  of  a  risk  event  occurring  and 
to  estimate  the  impact  (consequences)  if  it 
occurs.  The  combination  of  these  two  fac¬ 
tors  determines  severity.  For  example,  an 
event  with  a  low  probability  of  occurring, 
yet  with  severe  consequences,  may  be  a 
candidate  for  handling.  Conversely,  an 
event  with  a  high  probability  of  happen¬ 
ing,  but  the  consequences  of  which  do  not 
affect  a  program,  may  be  acceptable  and 
require  no  handling. 

To  reduce  uncertainty  and  apply  the  defini¬ 
tion  of  risk  to  acquisition  programs,  PMs 
must  be  familiar  with  the  t^es  of  acquisi¬ 


tion  risks,  understand  risk  terminology,  and 
know  how  to  measure  risk.  These  topics  are 
addressed  in  the  next  several  sections. 

2.4.1  Characteristics  of 
Acquisition  Risk 

Acquisition  programs  tend  to  have  numer¬ 
ous,  often  interrelated,  risks.  They  are  not 
always  obvious;  relationships  may  be  ob¬ 
scure;  and  they  may  exist  at  aU  program  lev¬ 
els  throughout  the  life  of  a  program.  Risks 
are  in  the  PMO  (program  plans,  etc.);  in  sup¬ 
port  provided  by  other  Government  agen¬ 
cies;  in  threat  assessment;  and  in  prime  con¬ 
tractor  processes,  engineering  and  manufac¬ 
turing  processes,  and  technology.  The  inter¬ 
relationship  among  risk  events  may  cause  an 
increase  in  one  because  of  the  occurrence  of 
another.  For  example,  a  slip  in  schedule  for 
an  early  test  event  may  adversely  impact  sub¬ 
sequent  tests,  assuming  a  fixed  period  of  test 
time  is  available. 

Another  important  risk  characteristic  is  the 
time  period  before  a  risk  future  event  occurs; 
because  time  is  critical  in  determining  risk¬ 
handling  options.  If  an  event  is  imminent, 
the  PMO  must  resort  to  crisis  management. 
An  event  that  is  far  enough  in  the  future  to 
allow  management  actions  may  be  control¬ 
lable.  The  goal  is  to  avoid  the  need  to  revert 
to  problem  solving  by  managing  risk. 

An  event's  probability  of  occurrence  and 
consequences  may  change  as  the  develop¬ 
ment  process  proceeds  and  information 
becomes  available.  Therefore,  throughout 
the  development  phase,  PMOs  should  re¬ 
evaluate  known  risks  on  a  periodic  basis 
and  examine  the  program  for  new  risks. 

2.4.2  Program  Products,  Processes,  Risk 
Areas,  and  Risk  Events 

Program  risk  includes  all  risk  events  and 
their  relationships  to  each  other.  It  is  a  top- 
level  assessment  of  impact  to  the  program 
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when  all  risk  events  at  the  lower  levels  of 
the  program  are  considered.  Program  risk 
may  be  a  roll-up  of  all  low-level  events; 
however,  most  likely,  it  is  a  subjective 
evaluation  of  the  known  risks  by  the  PMO, 
based  on  the  judgment  and  experience  of 
experts.  Identifying  program  risk  is  worth¬ 
while  because  it  forces  the  PMO  to  consider 
relationships  among  all  risks  and  may 
identify  potential  areas  of  concern  that 
would  have  otherwise  been  overlooked. 
One  of  the  greatest  strengths  of  a  formal, 
continuous  risk  management  process  is  the 
proactive  quest  to  identify  risk  events  for 
handling  and  the  reduction  of  uncertainty 
that  results  from  handling  actions. 

A  program  office  has  continuous  demands 
on  its  time  and  resources.  It  is,  at  best,  diffi¬ 
cult,  and  probably  impossible,  to  assess  ev¬ 
ery  potential  area  and  process.  To  manage 
risk,  the  PMOs  should  focus  on  the  critical 
areas  that  could  affect  the  outcome  of  their 
programs.  Work  Breakdown  Structure 
(WBS)  product  and  process  elements  and  in¬ 
dustrial  engineering  and  manufacturing  pro¬ 
cesses  contain  most  of  the  significant  risk 
events.  Risk  events  are  determined  by  ex¬ 
amining  each  WBS  element  and  process  in 
terms  of  sources  or  areas  of  risk.  Following 
are  some  t5q)ical  risk  areas: 

•  Threat.  The  sensitivity  of  the  program 
to  uncertainty  in  the  threat  description,  the 
degree  to  which  the  system  design  would 
have  to  change  if  the  threat's  parameters 
change,  or  the  vulnerability  of  the  program 
to  foreign  intelligence  collection  efforts 
(sensitivity  to  threat  countermeasure). 

•  Requirements.  The  sensitivity  of  the 
program  to  uncertainty  in  the  system  de¬ 
scription  and  requirements  except  for  those 
caused  by  threat  uncertainty. 

•  Design.  The  abihty  of  the  system  con¬ 
figuration  to  achieve  the  program's  engineer¬ 
ing  objectives  based  on  the  available  tech¬ 


nology,  design  tools,  design  maturity,  etc. 

•  Test  and  Evaluation.  The  adequacy  and 
capability  of  the  test  and  evaluation  program 
to  assess  attainment  of  significant  perfor¬ 
mance  specifications  and  determine  whether 
the  systems  are  operationally  effective  and 
suitable. 

•  Modeling  and  Simulation  (M&S).  The 
adequacy  and  capability  of  M&S  to  support 
all  phases  of  a  program  using  verified,  valid, 
and  accredited  M&S. 

•  Technology.  The  degree  to  which  the 
technology  proposed  for  the  program  has 
been  demonstrated  as  capable  of  meeting  all 
of  the  program's  objectives. 

•  Logistics.  The  ability  of  the  system  con¬ 
figuration  to  achieve  the  program's  logistics 
objectives  based  on  the  system  design,  main¬ 
tenance  concept,  support  system  design,  and 
availability  of  support  resources. 

•  Production/Facilities.  The  ability  of 
the  system  configuration  to  achieve  the 
program's  production  objectives  based  on 
the  system  design,  manufacturing  pro¬ 
cesses  chosen,  and  availability  of  manufac¬ 
turing  resources. 

•  Concurrency.  The  sensitivity  of  the 
program  to  uncertainty  resulting  from  the 
combining  or  overlapping  of  life-cycle 
phases  or  activities. 

•  Capability  of  Developer.  The  ability 
of  the  developer  to  design,  develop,  and 
manufacture  the  system.  The  contractor 
should  have  the  experience,  resources,  and 
knowledge  to  produce  the  system. 

•  Cost/Fimding.  The  ability  of  the  sys¬ 
tem  to  achieve  the  program's  life-cycle  sup¬ 
port  objectives.  This  includes  the  effects  of 
budget  and  affordability  decisions  and  the 
effects  of  inherent  errors  in  the  cost  estimat¬ 
ing  technique(s)  used  (given  that  the  techni¬ 
cal  requirements  were  properly  defined). 
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•  Management.  The  degree  in  which 
program  plans  and  strategies  exist  and  are 
realistic  and  consistent.  The  Government's 
acquisition  team  should  be  qualified  and 
sufficiently  staffed  to  manage  the  program. 

•  Schedule.  The  adequacy  of  the  time 
allocated  for  performing  the  defined  de¬ 
velopmental  tasks.  This  factor  includes  the 
effects  of  programmatic  schedule  decisions, 
the  inherent  errors  in  the  schedule  estimat¬ 
ing  technique  used,  and  external  physical 
constraints. 

Critical  risk  processes  are  the  developer's  en¬ 
gineering  and  manufacturing  processes 
which,  historically,  have  caused  the  most  dif¬ 
ficulty  during  the  development  phases  of  ac¬ 
quisition  programs.  These  processes  are  de¬ 
sign,  test,  production,  facilities,  logistics, 
and  management.  Three  of  these  processes 
are  included  in  the  critical  risk  areas  and  are 
addressed  separately  to  emphasize  that  they 
focus  on  processes.  DoD  4245.7-M,  Transi¬ 
tion  from  Development  to  Production,  describes 
them  using  templates.  See  Figure  2-2  for  an 
example  of  the  template  for  product  devel¬ 
opment.  The  templates  are  the  result  of  a 
Defense  Science  Board  task  force,  composed 
of  Government  and  industry  experts,  who 
identified  engineering  processes  and  control 
methods  to  minimize  risk  in  both  Govern¬ 
ment  and  industry.  The  task  force  defined 
these  critical  events  in  terms  of  the  templates, 
which  are  briefly  discussed  later.  The  figure 
also  shows  funding  as  a  process  that,  unlike 
others,  is  a  Government  process. 

Additional  areas,  such  as  manpower,  envi¬ 
ronmental  impact,  systems  safely  and  health, 
and  systems  engineering,  that  are  analyzed 
during  program  plan  development  provide 
indicators  for  additional  risk.  The  PMO 
should  consider  these  areas  for  early  assess¬ 
ment  since  failure  to  do  so  could  cause  dire 
consequences  in  the  program's,  latter  phases. 


In  addition,  PMs  should  address  the  un¬ 
certainty  associated  with  security —  an  area 
sometimes  overlooked  by  developers  but 
addressed  in  the  Acquisition  System  Pro¬ 
tection  (ASP)  section  of  the  Deskbook  and 
Air  Force  Pamphlet  ASPWG  PH-1,  Acqui¬ 
sition  System  Protection  Program  Work  Book, 
September  1994,  and  in  the  Deskbook. 
Defense  Systems  Management  College, 
Program  Management  Teaching  Note,  Ac¬ 
quisition  Program  Protection,  12  January 
1998.  However,  in  addition  to  the  guid¬ 
ance  given  there,  PMs  must  recognize  that, 
in  the  past,  classified  programs  have  expe¬ 
rienced  difficulty  in  access,  facilities,  clear¬ 
ances,  and  visitor  control.  Failure  to  man¬ 
age  these  aspects  of  a  classified  program 
could  affect  schedules. 

2.5  RISK  PLANNING 

2.5.1  Purpose  of  Risk  Plans 

Risk  planning  is  the  detailed  formulation 
of  a  program  of  action  for  the  management 
of  risk.  It  is  the  process  to: 

•  Develop  and  document  an  organized, 
comprehensive,  and  interactive  risk  man¬ 
agement  strategy. 

•  Determine  the  methods  to  be  used  to 
execute  a  PM's  risk  management  strategy. 

•  Plan  for  adequate  resources. 

Risk  planning  is  iterative  and  includes  the 
activities  to  assess,  control,  monitor,  and 
document  the  risk  associated  with  a  program. 
The  result  is  the  Risk  Management  Plan 
(RMP). 

2.5.2  Risk  Planning  Process 

The  PMO  should  periodically  review  the 
plan  and  revise  it,  if  necessary.  Some  events 
such  as:  (1)  a  change  in  acquisition  strategy 

(2)  preparation  for  a  major  decision  point, 

(3)  technical  audits  and  reviews,  (4)  an  up¬ 
date  of  other  program  plans,  and  (5)  prepa- 
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Areas  and  Templates 


ration  for  a  Program  Objective  Memorandum 
(POM)  submission  may  drive  the  need  to  up¬ 
date  an  existing  plan. 

Planning  begins  by  developing  and  docu¬ 
menting  a  risk  management  strategy.  Early 
efforts  establish  the  purpose  and  objective, 
assign  responsibilities  for  specific  areas,  iden¬ 
tify  additional  technical  expertise  needed, 
describe  the  assessment  process  and  areas  to 
consider,  delineate  procedures  for  consider¬ 
ation  of  handling  options,  define  a  rating 
scheme,  dictate  the  reporting  and  documen¬ 
tation  needs,  and  establish  report  require¬ 
ments  and  monitoring  metrics.  This  plan¬ 
ning  should  also  address  evaluation  of  the 


capabilities  of  potential  sources  as  well  as 
early  industry  involvement  and  program. 

The  PM's  strategy  to  manage  risk  provides 
the  program  team  with  direction  and  basis 
for  planning.  Initially  formalized  during 
a  program's  Concept  Exploration  Phase 
and  updated  for  each  subsequent  program 
phase,  the  strategy  should  be  reflected  in 
the  program's  acquisition  strategy,  which 
with  requirement  and  threat  documents, 
known  risks,  and  system  and  program 
characteristics  are  sources  of  information 
for  PMO  use  to  devise  a  strategy  and  be¬ 
gin  developing  a  risk  management  plan. 
Since  the  program's  risks  are  affected  by 
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the  Government  and  contractor  team's  abil¬ 
ity  to  develop  and  manufacture  the  system, 
industry  can  provide  valuable  insight  into 
this  area  of  consideration. 

The  plan  is  the  roadmap  that  tells  the  Gov¬ 
ernment  and  contractor  team  how  to  get 
from  where  the  program  is  today  to  where 
the  PM  wants  it  to  be  in  the  future.  The 
key  to  writing  a  good  plan  is  to  provide 
the  necessary  information  so  the  program 
team  knows  the  objectives,  goals,  and  the 
PMC's  risk  management  process.  Since  it 
is  a  map,  it  may  be  specific  in  some  areas, 
such  as  the  assignment  of  responsibilities 
for  Government  and  contractor  partici¬ 
pants  and  definitions,  and  general  in  other 
areas  to  allow  users  to  choose  the  most  ef¬ 
ficient  way  to  proceed.  For  example,  a  de¬ 
scription  of  techniques  that  suggests  sev¬ 
eral  methods  for  evaluators  to  use  to  as¬ 
sess  risk  is  appropriate,  since  every  tech¬ 
nique  has  advantages  and  disadvantages 
depending  on  the  situation. 

Appendix  B  contains  an  example  of  a  risk 
plan  and  a  summary  of  the  format  is  shown 
in  Figure  2-3. 


In  a  decentralized  PMO  risk  management 
organization,  the  program's  risk  manage¬ 
ment  coordinator  may  be  responsible  for 
risk  management  planning.  See  Sections 
4.4,  Risk  Management  Organizations,  and 
5.3,  Risk  Planning  Techniques. 

2.6  RISK  ASSESSMENT 

2.6.1  Purpose  of  Risk  Assessments 

The  primary  objective  of  assessments  is  to 
identify  and  analyze  program  risks  so  that 
the  most  critical  among  them  may  be  con¬ 
trolled.  Assessments  are  factors  that  man¬ 
agers  should  consider  in  setting  technical, 
schedule,  and  cost  objectives  because  they 
provide  an  indication  of  the  hkelihood  of 
achieving  the  desired  outcomes. 

2.6.2  Risk  Assessment  Process 

Risk  assessment  is  the  problem  definition 
stage  of  management  that  identifies,  ana¬ 
lyzes,  and  quantifies  program  events  in 
terms  of  probability  and  consequences. 
The  results  form  the  basis  for  most  risk 
management  actions.  It  is  probably  the 
most  difficult  and  time-consuming  part  of 
the  management  process.  There  are  no 
quick  answers  or  shortcuts.  Tools  are  avail- 
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Figure  2-3.  A  Risk  Management  Pian  Format 
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able  to  assist  evaluators  in  assessing  risk, 
but  none  are  totally  suitable  for  any  pro¬ 
gram  and  may  be  misleading  if  the  user 
does  not  understand  how  to  apply  them 
or  interpret  the  results.  Despite  its  com¬ 
plexity,  risk  assessment  is  one  of  the  most 
important  phases  of  the  risk  process  be¬ 
cause  the  caliber  and  quality  of  assessments 
determine  the  effectiveness  of  a  manage¬ 
ment  program. 

The  components  of  assessment,  identifica¬ 
tion  analysis  and  prioritization,  are  per¬ 
formed  sequentially  with  identification 
being  the  first  step. 

Risk  identification  begins  by  compiling  the 
program's  risk  events.  PMOs  should  ex¬ 
amine  and  identify  program  events  by  re¬ 
ducing  them  to  a  level  of  detail  that  per¬ 
mits  an  evaluator  to  understand  the  sig¬ 
nificance  of  any  risk  and  identify  its  causes, 
i.e.,  risk  drivers.  This  is  a  practical  way  of 
addressing  the  large  and  diverse  number 
of  potential  risks  that  often  occur  in  acqui¬ 
sition  programs.  For  example,  a  WBS  level 
4  or  5  element  may  be  made  up  of  several 
risk  events  associated  with  a  specification 
or  function,  e.g.,  failure  to  meet  turbine 
blade  vibration  requirements  for  an  engine 
turbine  design. 

Risk  events  are  best  identified  by  examin¬ 
ing  each  WBS  product  and  process  element 
in  terms  of  the  sources  or  areas  of  risk,  as 
previously  described  in  Paragraph  2.4.2. 

Risks  are  those  events  that  evaluators,  (after 
examining  scenarios,  WBS,  or  processes), 
determine  would  adversely  affect  the 
program.  Evaluators  may  initially  rank 
events  by  probability  of  occurrence  based  on 
a  judgment  of  consequence  before  beginning 
analysis  to  focus  on  those  most  critical. 

Risk  analysis  is  a  technical  and  systematic 
process  to  examine  identified  risks,  isolate 


causes,  determine  the  relationship  to  other 
risks,  and  express  the  impact  in  terms  of 
probability  and  consequences. 

In  practice,  the  distinction  between  risk 
identification  and  risk  analysis  is  often 
blurred  because  there  is  some  risk  analy¬ 
sis  that  occurs  during  the  identification 
process.  For  example,  if,  in  the  process  of 
interviewing  an  expert,  a  risk  is  identified, 
it  is  logical  to  pursue  information  on  the 
probability  of  it  occurring,  the  conse¬ 
quences,  the  time  associated  with  the  risk 
(i.e.,  when  it  might  occur),  and  possible 
ways  of  dealing  with  it.  The  latter  actions 
are  part  of  risk  analysis  and  risk  handling, 
but  often  begin  during  risk  identification. 

Prioritization  is  the  ranking  of  risk  events 
to  determine  the  order  of  importance.  It 
serves  as  the  basis  for  risk-handling  actions. 

Integrated  Product  Teams  (IPTs)  typically 
perform  risk  assessments  in  a  decentralized 
risk  management  organization  as  de¬ 
scribed  in  paragraph  4.4.  If  necessary,  the 
team  may  be  augmented  by  people  from 
other  program  areas  or  outside  experts. 
Paragraph  5.4,  Risk  Assessment  Tech¬ 
niques,  elaborates  on  this  for  each  of  the 
described  assessment  techniques. 

2.6.3  Timing  of  Risk  Assessments 

The  assessment  process  begins  during  the 
last  half  of  Phase  0,  Concept  Exploration, 
and  continues  throughout  the  subsequent 
phases.  The  PMO  should  continually  re¬ 
assess  the  program  at  increasing  levels  of 
detail  as  the  program  progresses  through 
the  acquisition  phases  and  more  informa¬ 
tion  becomes  available.  There  are,  how¬ 
ever,  times  when  events  may  require  new 
assessments,  i.e.,  a  major  change  in  the  ac¬ 
quisition  strategy.  Paragraph  2.5.2  lists 
other  events  that  could  cause  risk  assess¬ 
ments  to  be  performed. 
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2.6.4  Conducting  Risk  Assessments 

There  is  no  standard  approach  to  assess¬ 
ing  risk  because  methods  vary  according 
to  the  technique  employed,  the  phase  of  the 
program,  and  the  nature  of  the  program 
itself;  however,  some  top-level  actions  are 
t5rpically  common  to  all  methods.  They  are 


grouped  in  Figure  2-4  into  pre-risk  assess¬ 
ment  activities,  risk  identification  activities, 
and  risk  analysis  activities. 

2.6.4.1  Pre-Risk  Assessment  Activities.  The 
risk  management  plan  may  describe  the  ac¬ 
tions  that  compose  this  activity.  Typically,  a 


Figure  2-4.  Risk  Assessment 
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program-level  IPX  may  conduct  a  quick-look 
assessment  of  the  program  to  identify  the 
need  for  technical  experts  (who  are  not  part 
of  the  team)  and  to  examine  areas  that  ap¬ 
pear  most  likely  to  contain  risk.  The 
program's  risk  coordinator,  or  an  outside  ex¬ 
pert,  may  train  the  IPTs,  focusing  on  the 
program's  risk  strategy,  definitions,  sug¬ 
gested  techniques,  documentation,  and  re¬ 
porting  requirements.  Paragraph  4.9,  Risk 
Management  Training,  provides  some  sug¬ 
gestions  for  training. 

2.6.4^  Risk  Identification  Activity.  To  iden¬ 
tify  risk  events,  IPTs  should  break  down  pro¬ 
gram  elements  to  a  level  where  they,  or  sub¬ 
ject  matter  experts,  can  perform  valid  assess¬ 
ments.  The  information  necessary  to  do  this 
varies  according  to  the  phase  of  the  program. 
EHiring  the  early  phases,  requirement,  tiireat 
documents,  and  acquisition  plans  may  be  the 
only  program-specific  data  available.  They 
should  be  analyzed  to  identify  events  that 
may  have  adverse  consequences.  A  useful 
initial  identification  exercise  is  to  perform  a 
mission  profile  for  the  system  as  suggested 
in  DoD4245.7-M,  Transition  from  Development 
to  Production.  Using  this  methodology,  the 


developer  creates  a  functional  and  environ¬ 
mental  profile  for  the  system  and  examines 
the  low-level  requirements  that  the  system 
must  meet  to  satisfy  its  mission  requirements. 
The  IPTs  may  then  study  these  requirements 
to  determine  which  are  critical.  For  example, 
in  an  aircraft  profile,  it  may  be  apparent  that 
high  speed  is  critical.  If  the  speed  require¬ 
ment  is  close  to  that  achieved  by  existing  air¬ 
craft,  this  may  not  be  a  concern.  However,  if 
the  speed  is  greater  than  that  achieved  by 
toda5^s  aircraft,  it  may  be  a  critical  risk  area. 
Since  aircraft  speed  depends,  among  other 
things,  on  weight  and  engine  thrust,  it  would 
be  desirable  to  enlist  the  help  of  a  materials 
expert  to  address  weight  and  an  engine  ex¬ 
pert  to  assess  engine-associated  risk. 

Another  method  of  decomposition  is  to 
create  a  WBS  as  early  as  possible  in  a  pro¬ 
gram.  Figure  2-5  is  a  simple  example  of  a 
decomposition  based  on  the  WBS  for  an 
aircraft.  The  figure  shows  an  important 
requirement  of  the  decomposition  process, 
the  establishment  of  goals,  e.g.,  don't  ex¬ 
ceed  the  weight  budget  or  objective.  Fur¬ 
ther  it  is  beneficial  at  this  time  to  estimate 
a  cost  distribution  for  each  risk  event. 
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During  decomposition,  risks  are  identified 
from  experience,  brainstorming,  lessons 
learned  fern  similar  programs,  and  guidance 
contained  in  the  risk  management  plan.  The 
examination  of  each  event  is  an  exploratory 
exercise  to  identify  the  critical  risks.  The  in¬ 
vestigation  may  show  that  risks  are  inter-re¬ 
lated.  For  example,  the  weight  of  an  aircraft 
affects  its  speed,  but  also  impacts  the  pay- 
load,  range,  and  fuel  requirements.  These 
have  design  and  logistics  consequences  and 
may  even  affect  the  number  of  aircraft  that 
must  be  procured  to  meet  objectives. 

Critical  risks  need  to  be  documented  as 
specified  in  the  risk  management  plan  and 
may  include  the  scenario  that  causes  the 
risk,  planned  management  controls  and  ac¬ 
tions,  etc.  It  may  also  contain  an  initial  as¬ 
sessment  of  the  consequences  to  focus  the 
risk  assessment  effort. 

2.6.4.3  Risk  Analysis  Activity.  Analysis 
begins  with  a  detailed  study  of  the  critical 
risks  that  have  been  identified.  The  objec¬ 
tive  is  to  gather  enough  information  about 
the  risks  to  judge  the  probability  of  occur¬ 
rence  and  the  impact  on  performance,  cost, 
and  schedule  if  the  risk  occurs. 

Impact  assessments  are  normally  subjective 
and  based  on  detailed  information  that 


•  Comparisons  with  similar  systems, 

•  Relevant  lessons-learned  studies, 

•  Experience, 

•  Results  from  tests  and  prototype 
development, 

•  Data  from  engineering  or  other  models, 

•  Specialist  and  expert  judgments, 

•  Analysis  of  plans  and  related 
documents, 

•  Modeling  and  simulation, 

•  Sensitivity  analysis  of  alternatives. 

Depending  on  the  particular  technique  and 
the  risk  being  analyzed,  some  supporting 
analysis  may  be  necessary,  i.e.,  analysis  of 
contractor  processes,  such  as  design,  engi¬ 
neering,  fault  tree  analysis,  engineering 
models,  simulation,  etc.  Analyses  provide 
the  basis  for  subjective  assessments. 

A  critical  aspect  of  risk  analysis  is  data  col¬ 
lection.  Two  primary  sources  of  data  are 
interviews  of  subject  matter  experts  and 
analogy  comparisons  with  similar  systems. 
Paragraph  5.4  contains  a  technique  for  col¬ 
lecting  both  types  of  data  for  use  in  sup¬ 
port  of  the  techniques  listed  in  Table  2-1. 
Periodically,  sets  of  risks  need  to  be  priori¬ 
tized  in  preparation  for  risk  handling,  and 


may  come  from: 


Table  2-1 .  Risk  Assessment  Approaches 


Risk  Assessment  Technique 

Applicable  Acquisition  Phases 

Applicable  Risk  Areas  & 
Processes 

Process  (DoD  4265.7-M)  Risk 
Assessment 

All  phases 

All  critical  risks  processes 

Plan  Evaluation  Risk 

Identification 

All  phases 

Program  plans  and  critical 
communication  with  the 
developer 

Product  (WBS)  Risk  Assessment 

All  phases  starting  with  the 
completion  of  the  Contract  WBS 

All  critical  risk  areas  except 
threat,  requirements,  cost  and 
schedule 

Cost  Risk  Assessment 

All  phases 

Cost  critical  risk  area 

Schedule  Risk  Assessment 

All  phases 

Schedule  critical  risk  area 
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aggregated  to  support  program  manage¬ 
ment  reviews.  Paragraph  5.5,  Risk 
Prioritization,  describes  methods  for  ac¬ 
complishing  this. 

2.6.4.4  Risk  Assessment  by  Risk  Category. 

Each  risk  category,  e.g.,  technical,  cost,  and 
schedule,  includes  a  core  set  of  assessment 
tasks  and  is  related  to  the  other  two  cat¬ 
egories.  This  relationship  requires  support¬ 
ive  analysis  among  areas  to  ensure  the  in¬ 
tegration  of  the  assessment  process.  For 
example,  a  technical  assessment  probably 
should  include  a  cost  and  schedule  analy¬ 
sis  in  determining  the  technical  risk  impact. 
The  results  of  the  assessments,  normally 
conducted  by  IPTs  follow: 

Technical  Assessment 

•  Provides  technical  foundation, 

•  Identifies  and  describes  program  risks, 
i.e.,  threat,  technology,  design,  manufactur¬ 
ing,  etc., 

•  Prioritizes  risks  with  relative  or  quan¬ 
tified  weight  for  program  impact, 

•  Analyzes  risks  and  relates  them  to 
other  internal  and  external  risks, 

•  Quantifies  associated  program  activi¬ 
ties  with  both  time  duration  and  resources, 

•  Quantifies  inputs  for  schedule  assess¬ 
ment  and  cost  estimate, 

•  Documents  technical  basis  and  risk 
definition  for  the  risk  assessment. 

Schedule  Assessment 

•  Evaluates  baseline  schedule  inputs, 

•  Incorporates  technical  assessment  in¬ 
puts  to  program  schedule  model, 

•  Evaluates  impacts  to  program  sched¬ 
ule  based  on  technical  team  assessment, 

•  Performs  schedule  analysis  on  pro¬ 
gram  integrated  master  schedule. 


•  Quantifies  schedule  excursions  reflect¬ 
ing  effects  of  cost  risks,  including  resource 
constraints, 

•  Provides  Government  schedule  as¬ 
sessment  for  cost  analysis  and  fiscal  year 
planning, 

•  Reflects  technical  foundation,  activity 
definition,  and  inputs  from  technical  and 
cost  areas, 

•  Documents  schedule  basis  and  risk 
impacts  for  the  risk  assessment. 

Cost  Estimate  and  Assessment 

•  Builds  on  technical  and  schedule  as¬ 
sessment  results, 

•  Translates  technical  and  schedule  risks 
into  cost, 

•  Derives  cost  estimate  by  integrating 
technical  assessment  and  schedule  risk 
impacts  to  resources, 

•  Establishes  budgetary  requirements 
consistent  with  fiscal  year  planning, 

•  Determines  if  the  phasing  of  funds 
supports  technical  and  acquisition  ap¬ 
proach, 

•  Provides  program  cost  excursions 
from: 

-  Nearterm  budget  execution  impacts, 

-  External  budget  changes  and  con¬ 
straints. 

•  Documents  cost  basis  and  risk  impacts. 

2.6.5  Risk  Rating 

Ratings  are  an  indication  of  the  potential 
impact  of  risks  on  a  program;  they  are  a 
measure  of  the  likelihood  of  an  event  oc¬ 
curring  and  the  consequences  of  the  event. 
They  are  often  expressed  as  high,  moder¬ 
ate,  and  low. 

A  group  of  experts,  who  are  familiar  with 
each  risk  area  (e.g.,  design/  logistics,  produc- 
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tion,  etc.)  and  product  WBS  element  risk  rat¬ 
ings,  are  best  qualified  to  determine  risk  rat¬ 
ings.  They  should  identify  rating  criteria  for 
review  by  the  PMO,  who  includes  them  in 
the  Risk  Management  Plan.  In  most  cases, 
the  criteria  will  be  based  on  the  experience 
of  the  experts,  as  opposed  to  mathematically 
derived  and  should  establish  levels  of  likeli¬ 
hood  and  consequences  that  wiU  provide  a 
range  of  possibilities  large  enough  to  distin¬ 
guish  differences  in  risk  ratings.  At  the  pro¬ 
gram  level,  consequences  should  be  ex¬ 
pressed  in  terms  of  impact  on  performance, 
cost,  and  schedule.  Tables  2-2  and  2-3  are 
examples  of  likelihood  and  consequence  cri¬ 
teria,  and  Table  2-4  contains  an  example  of 
overall  risk  rating  criteria,  which  considers 
both  likelihood  and  consequences. 


Table  2-2.  Likelihood  Criteria 
(Example) 


Level 

What  is  the  Likelihood  the  Risk 
Event  Will  Happen? 

a 

Remote 

b 

Unlikely 

c 

Likely 

d 

Highly  likely 

e 

Near  certainty 

Using  these  risk  ratings.  Program  Manag¬ 
ers  can  identify  events  requiring  priority 
management  (high  or  moderate  risk  likeli¬ 
hood  or  consequences).  Risk  ratings  also 
help  to  identify  the  areas  that  should  be 
reported  within  and  outside  the  PMO,  e.g., 
milestone  decision  reviews.  Thus,  it  is  im¬ 
portant  that  the  ratings  be  portrayed  as  ac¬ 
curately  as  possible. 

A  simple  method  of  representing  the  risk 
rating  for  risk  events  is  shown  in  Fi^re 
2-6.  In  this  example,  the  PM  has  defined 
levels  high,  moderate,  and  low  for  the  vari¬ 
ous  combinations  of  likelihood  and  conse¬ 
quences. 

There  is  a  common  tendency  to  attempt  to 
develop  a  single  number  to  portray  the  risk 
associated  with  a  particular  event.  This  ap¬ 
proach  may  be  suitable  if  both  likelihood 
(probability)  and  consequences  have  been 
accurately  quantified  using  compatible  car¬ 
dinal  scales.  In  such  a  case,  mathematical 
manipulation  of  the  values  may  be  mean¬ 
ingful  and  may  provide  some  quantitative 
basis  for  the  ranking  of  risks.  In  many 
cases,  however,  risk  scales  are  expressed 
on  an  ordinal  scale  such  as  first,  second, 
etc.,  or  level  a,  b,  or  c,  that  reflect  relative 
standing  between  ratings  and  not  absolute 


Table  2-3.  Consequences  Criteria  (Example) 


Level 

Given  the  Risk  is  Realized,  What  is  the  Magnitude  of  the  Impact? 

Performance 

Schedule 

Cost 

a 

Minimal  or  no  impact 

Minimal  or  no  impact 

Minimum  or  no  impact 

b 

Acceptable  with  some 
reduction  in  margin 

Additional  resources 
required;  able  to  meet  need 
dates 

<5% 

c 

Acceptable  with  significant 
reduction  in  margin 

Minor  slip  in  key 
milestones:  not  able  to 
meet  need  date 

5-7% 

d 

Acceptable:  no  remaining 
margin 

Major  slip  in  key  milestone 
or  critical  path  impacted 

7-10% 

e 

Unacceptable 

Can’t  achieve  key  team  or 
major  program  milestone 

>10% 
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Table  2-4.  Overall  Risk  Rating  Criteria 
(Example) 


Rating 

Description 

High 

Moderate 

Low 

Major  disruption  likely 
Some  disruption 
Minimum  impact 

■D 

8 

x: 

1 


e 

L 

M 

H 

H 

H 

d 

L 

M 

M 

H 

H 

c 

L 

M 

M 

M 

H 

b 

L 

L 

L 

M 

M 

a 

L 

L 

L 

L 

M 

1 

2 

3 

4 

5 

Consequence 


Figure  2-6.  Overall  Risk  Rating 
(Example) 

numerical  differences.  Any  mathematical  op¬ 
erations  performed  on  ordinal  scales,  or  a  com¬ 
bination  of  ordinal  and  cardinal  scales,  can  pro¬ 
vide  information  that  will  at  best  be  mislead¬ 
ing,  if  not  completely  meaningless,  possibly  re¬ 
sulting  in  erroneous  risk  ratings. 


One  way  to  avoid  this  situation  is  to  sim¬ 
ply  show  each  risk  event's  likelihood  and 
consequences  separately,  with  no  attempt 
to  combine  multiple  risks.  Other  factors 
that  may  significantly  contribute  to  the  risk 
rating  or  prioritization  of  risk  events,  such 
as  time  sensitivity  or  resource  availability, 
can  also  be  shown.  The  prioritization 


should  be  done  based  on  expert  opinion 
and  experience.  Table  2-5  provides  a 
sample  format  for  presenting  risk  ratings. 

2.7  RISK  HANDLING 

2.7.1  Purpose  of  Risk  Handling 

Risk  handling  includes  specific  methods  and 
techniques  to  deal  with  known  risks  and  a 
schedule  for  accomplishing  tasks,  identifies 
who  is  responsible  for  the  risk  area,  and  pro¬ 
vides  an  estimate  of  the  cost  and  schedule 
impact,  if  any.  It  involves  planning  and  ex¬ 
ecution  with  the  objective  of  controlling  risks 
at  an  acceptable  levels.  The  IPTs  that  assess 
risk  should  begin  the  process  to  identify  and 
evaluate  handling  approaches  to  propose  to 
the  PM,  who  selects  the  appropriate  ones  for 
implementation. 

2.7.2  Risk  Handling  Process 

The  risk-handling  phase  must  be  compatible 
with  the  risk  management  plan  and  any  ad¬ 
ditional  guidance  the  PM  provides.  Para¬ 
graph  5.3  describes  a  technique  that  concen¬ 
trates  on  planning.  A  critical  part  planning 
involves  refining  and  selecting  of  the  most 
appropriate  handling  options. 

The  IPTs  that  evaluate  the  handling  options 
may  use  the  following  criteria  as  a  starting 
point  for  assessment: 

•  Can  the  option  be  feasibly  imple¬ 
mented  and  still  meet  the  user's  needs? 

•  What  is  the  expected  effectiveness  of 
the  handling  option  in  reducing  program 
risk? 


Table  2-5.  Risk  Ratings  (Example) 


Priority 

Area/ 

Process 

Location 

Title 

Likeli¬ 

hood 

Conse¬ 

quence 

Time 

Constraints 

1 

Design 

WBS  3.1 

Design 

completeness 

High 

High 

1-2  months 

2 

3 
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•  Is  the  option  affordable  in  terms  of  dol¬ 
lars  and  other  resources  (e.g.,  use  of  criti¬ 
cal  materials,  test  facilities,  etc.)? 

•  Is  time  available  to  develop  and  imple¬ 
ment  the  option,  and  what  effect  does  that 
have  on  the  overall  program  schedule? 

•  What  effect  does  the  option  have  on 
the  system's  technical  performance? 

Risk-handling  options  can  include  risk 
avoidance,  risk  control,  risk  transfer,  risk 
assumption. 

Risk  avoidance  involves  a  change  in  the 
concept,  requirements,  specifications,  and/ 
or  practices  that  reduce  risk  to  an  accept¬ 
able  level.  Simply  stated,  it  eliminates  the 
sources  of  high  risk  and  replaces  them  with 
a  lower  risk  solution  and  may  be  supported 
by  a  cost /benefit  analysis.  Generally,  this 
method  may  be  done  in  parallel  with  the 
up-front  requirements  analysis,  supported 
by  cost/requirement  trade  studies. 

Risk  control  does  not  attempt  to  eliminate 
the  source  of  the  risk  but  seeks  to  reduce 
or  mitigate  the  risks.  It  monitors  and  man¬ 
ages  the  risk  in  a  manner  that  reduces  the 
likelihood  of  its  occurrence  or  minimizes 
the  risk's  effect  on  the  program.  This  op¬ 
tion  may  add  to  the  cost  of  a  program.  A 
sampling  is  listed  below  of  the  types  of  risk 
control  actions  available  to  the  PMO. 

Paragraph  5.6.4  discusses  them  in  more  de¬ 
tail. 

•  Multiple  Development  Efforts.  Cre¬ 
ate  systems  that  meet  the  same  performance 
requirements. 

•  Alternative  Design.  Create  a  backup 
design  option  that  uses  a  lower  risk  ap¬ 
proach. 

•  Trade  Studies.  Arrive  at  a  balance  of 
engineering  requirements  in  the  design  of 
a  system. 


•  Early  Protot3q>ing.  Build  and  test  pro¬ 
totypes  early  in  the  system  development. 

•  Incremental  Development.  Design 
with  the  intent  of  upgrading  system  parts  in 
the  future. 

•  Technology  Maturation  Efforts. 
Normally,  technology  matiuation  is  used 
when  the  desired  technology  will  replace 
an  existing  technology  which  is  available 
for  use  in  the  system. 

•  Test-Anal)rze-and-Fix  (TAAF).  TAAF 
is  the  use  of  a  period  of  dedicated  testing 
to  identify  and  correct  deficiencies  in  a 
design. 

•  Robust  Design.  This  approach  uses 
advanced  design  and  manufacturing  tech¬ 
niques  that  promote  quality  through  de¬ 
sign. 

•  Design  of  Experiments.  This  engi¬ 
neering  tool  identifies  critical  design  fac¬ 
tors  that  are  sensitive,  therefore  potentially 
high  risk,  to  achieve  a  particular  user  re¬ 
quirement. 

•  Demonstration  Events.  Demonstra¬ 
tion  events  are  points  in  the  program  (nor¬ 
mally  tests)  that  determine  if  risks  are  be¬ 
ing  successfully  abated. 

•  Use  of  Mockups.  The  use  of  mock- 
ups,  especially  man-machine  interface 
mock-ups,  can  be  used  to  conduct  early  ex¬ 
ploration  of  design  options. 

•  Modeling/Simulation.  Modeling  and 
simulation  can  be  used  to  investigate  vari¬ 
ous  design  options  and  system  requirement 
levels. 

•  Key  Parameter  Control  Boards.  The 

practice  of  establishing  a  control  board  for 
a  parameter  may  be  appropriate  when  a 
particular  feature  (such  as  system  weight) 
is  crucial  to  achieving  the  overall  program 
requirements. 
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•  Process  Proofing.  Particular  pro¬ 
cesses,  especially  manufacturing  and  sup¬ 
port  processes,  are  critical  to  achieve  sys¬ 
tem  requirements. 

•  Manufacturing  Screening.  For  pro¬ 
grams  in  Engineering  and  Manufacturing 
Development  (EMD),  various  manufactur¬ 
ing  screens  (including  environmental  stress 
screening)  can  be  incorporated  into  test  ar¬ 
ticle  production  and  low  rate  initial  pro¬ 
duction  to  identify  deficient  manufactur¬ 
ing  processes. 

•  Reviews,  Walkthroughs,  and  Inspec¬ 
tions.  These  three  actions  can  be  used  to 
reduce  the  likelihood  and  potential  conse¬ 
quences  of  risks  through  timely  assessment 
of  actual  or  planned  events. 

•  Open  Systems.  Carefully  selected 
commercial  specifications  and  standards 
whose  use  can  result  in  lower  risks. 

•  Use  of  Standard  Items/Software  Re¬ 
use.  Use  of  existing  and  proven  hardware 
and  software,  where  applicable,  can  sub¬ 
stantially  reduce  risks. 

•  Two-Phase  Engineering  and  Manu¬ 
facturing  Development.  Incorporation  of 
a  formal  risk  reduction  phase  at  the  initial 
part  of  EMD. 

As  you  can  see,  there  are  numerous 
means  that  can  be  used  to  actively  control 
risks. 

Risk  Transfer.  This  action  may  reallocate 
risk  during  the  concept  development  and 
design  processes  from  one  part  of  the  sys¬ 
tem  to  another,  thereby  reducing  the  over¬ 
all  system  risk,  or  re-distributing  risks  be¬ 
tween  the  Government  and  the  prime  con¬ 
tractor  or  within  Government  agencies.  It 
is  an  integral  part  of  the  functional  analy¬ 
sis  process.  lOsk  transfer  is  a  form  of  risk 
sharing  and  not  risk  abrogation  on  the  part 
of  the  Government,  and  it  may  influence 


cost  objectives.  An  example  is  the  transfer 
of  a  function  from  hardware  implementa¬ 
tion  to  software  implementation  or  vice 
versa.  The  effectiveness  of  risk  transfer 
depends  on  the  use  of  successful  system 
design  techniques.  Modularity  and  func¬ 
tional  partitioning  are  two  design  tech¬ 
niques  that  support  risk  transfer.  In  some 
cases,  risk  transfer  may  concentrate  risk 
areas  in  one  area  of  the  design.  This  al¬ 
lows  management  to  focus  attention  and 
resources  on  that  area. 

Risk  Assumption.  Risk  assumption  is  an 
acknowledgment  of  the  existence  of  a  par¬ 
ticular  risk  situation  and  a  conscious  deci¬ 
sion  to  accept  the  associated  level  of  risk, 
without  engaging  in  any  special  efforts  to 
control  it.  However,  a  general  cost  and 
schedule  reserve  may  be  set  aside  to  deal 
with  any  problems  that  may  occur  as  a  re¬ 
sult  of  various  risk  assumption  decisions. 
This  method  recognizes  that  not  all  identi¬ 
fied  program  risks  warrant  special  han¬ 
dling;  as  such,  it  is  most  suited  for  those 
situations  that  have  been  classified  as  low 
risk.  The  key  to  successful  risk  assump¬ 
tion  is  twofold: 

•  Identify  the  resources  (time,  money, 
people,  etc.)  needed  to  overcome  a  risk  if  it 
materializes.  This  includes  identifying  the 
specific  management  actions  (such  as  re¬ 
testing,  additional  time  for  further  design 
activities)  that  may  occur. 

•  Ensure  that  necessary  administrative 
actions  are  taken  to  identify  a  management 
reserve  to  accomplish  those  management 
actions. 

Risk-handling  options  have  broad  cost 
implications.  The  magnitude  of  these  costs 
are  circumstance-dependent.  The  approval 
and  funding  of  handling  options  should  be 
part  of  the  process  that  establishes  the  pro¬ 
gram  cost  and  performance  goals.  The  se- 


19 


lected  handling  option  should  be  included 
in  the  program's  acquisition  strategy. 

Once  the  acquisition  strategy  includes  risk¬ 
handling  approaches,  the  PMO  can  derive 
the  schedule  and  identify  cost  impacts  to 
the  basic  program. 

2.8  RISK  MONITORING 

The  monitoring  process  systematically  tracks 
and  evaluates  the  effectiveness  of  risk-han¬ 
dling  actions  against  established  metrics. 
Monitoring  results  may  also  provide  a  basis 
for  developing  additional  handling  options 
and  identifying  new  risks.  The  key  to  the 
monitoring  process  is  to  establish  a  manage¬ 
ment  indicator  system  over  the  entire  pro¬ 
gram  that  the  PM  uses  to  evaluate  the  status 
of  the  program.  The  indicator  system  should 
be  designed  to  provide  early  warning  of  po¬ 
tential  problems  to  allow  management  ac¬ 
tions.  I^k  monitoring  is  not  a  problem-solv¬ 
ing  technique,  but  rather,  a  technique  to  con¬ 
trol  the  program  and  avoid  problems.  Some 
techniques  suitable  for  monitoring  can  be 
adapted  to  become  part  of  a  risk  indicator 
system: 

•  Test  and  Evaluation  (T&E).  A  well- 
defined  (T&E)  program  is  a  key  element 
in  monitoring  the  performance  of  selected 
risk-handling  options  and  developing  new 
risk  assessments. 

•  Earned  Value  (EV).  This  uses  stan¬ 
dard  DoD  cost/schedule  data  to  evaluate 
a  program's  cost,  schedule,  and  technical 
performance  in  an  integrated  fashion.  As 
such,  it  provides  a  basis  to  determine  if 
risk-handling  actions  are  achieving  their 
forecasted  results. 

•  Technical  Performance  Measurement 
(TPM).  TPM  is  a  product  design  assess¬ 
ment  which  estimates,  through  engineer¬ 
ing  analysis  and  tests,  the  values  of  essen¬ 
tial  performance  parameters  of  the  current 
design  as  effected  by  risk-  handling  actions. 


•  Program  Metrics.  These  are  formal, 
periodic  performance  assessments  of  the 
various  development  processes,  evaluating 
how  well  the  system  development  process 
is  achieving  its  objective.  This  technique 
can  be  used  to  monitor  corrective  actions 
that  emerged  from  an  assessment  of  the 
critical  risk  processes. 

•  Schedule  Performance  Monitoring. 
This  is  the  use  of  program  schedule  data 
to  evaluate  how  well  the  program  is  pro¬ 
gressing  to  completion. 

Paragraph  5.7  describes  several  monitor¬ 
ing  techniques,  e.g.,  earned  value. 

The  indicator  system  and  periodic  reassess¬ 
ments  of  program  risk  should  provide  the 
PMO  with  the  means  to  incorporate  risk 
management  into  the  overall  program 
management  structure. 

2.9  RISK  DOCUMENTATION 

A  primary  criteria  for  successful  manage¬ 
ment  is  formally  documenting  the  on-go- 
ing  risk  management  process.  This  is  im¬ 
portant  because: 

•  It  provides  the  basis  for  program  as¬ 
sessments  and  updates  as  the  program 
progresses. 

•  Formal  documentation  tends  to  ensure 
more  comprehensive  risk  assessments  than 
if  it  is  not  documented. 

•  It  provides  a  basis  for  monitoring  risk¬ 
handling  actions  and  verifying  the  results. 

•  It  provides  program  background  ma¬ 
terial  for  new  personnel. 

•  It  is  a  management  tool  for  the  execu¬ 
tion  of  the  program. 

•  It  provides  the  rationale  for  program 
decisions. 
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The  documentation  should  be  done  by  those 
responsible  for  planning  and  collecting  and 
analyzing  data,  i.e.,  IPT  level  in  most  cases. 

Risk  management  reports  vary  depending 
on  the  size,  nature,  and  phase  of  the  pro¬ 
gram.  Examples  of  some  risk  management 
documents  and  reports  that  may  be  useful 
to  a  Program  Manager  are: 

•  Management  plan, 

•  Risk  information  form, 

•  Risk  assessment  report, 

•  Risk  handling  priority  list, 

•  Risk  handling  plan  of  action, 

•  Aggregated  risk  hst, 

•  Risk  monitoring  documentation: 

-  Program  metrics, 

-  Technical  reports, 

-  Earned  value  reports, 

-  Watch  list, 

-  Schedule  performance  report, 

-  Critical  risk  processes  reports. 

Most  PMOs  can  devise  a  list  of  standard  re¬ 
ports  that  will  satisfy  their  needs  most  of  the 
time;  however,  since  there  will  always  be  a 
need  for  ad  hoc  reports  and  briefing  and  as¬ 
sessments,  it  is  advisable  to  store  risk  infor¬ 
mation  in  a  management  information  system 
(MIS).  This  allows  you  to  derive  standard 
reports  and  create  of  ad  hoc  reports,  as 
needed.  Paragraphs  4.8  and  5.8  discuss  an 
MIS  to  support  a  risk  management  program. 


Acquisition  reform  discourages  Govern¬ 
ment  oversight;  therefore,  formal  contrac¬ 
tor-produced  risk  documentation  may  not 
be  available  for  most  programs.  However, 
program  insight  is  encouraged,  and  PMOs 
can  obtain  information  about  program  risk 
from  contractor  internal  documentation 
such  as: 

•  Risk  Management  Policy  and  Proce¬ 
dures.  This  is  a  description  of  the 
contractor's  corporate  policy  for  the  man¬ 
agement  of  risk.  The  procedures  describe 
the  methods  for  risk  identification,  analy¬ 
sis,  handling,  monitoring,  and  documen¬ 
tation.  It  should  provide  the  baseline  plan¬ 
ning  document  for  the  contractor's  ap¬ 
proach  to  risk  management. 

•  Corporate  Policy  and  Procedures 
Documents.  Corporations  have  policy  and 
procedures  documents  that  address  the  func¬ 
tional  areas  that  are  critical  to  the  design,  en¬ 
gineering,  manufacture,  test  and  evaluation, 
quality,  configuration  control,  manufacture, 
etc.,  of  a  system.  These  documents  are  based 
on  what  the  company  perceives  as  best  prac¬ 
tices,  and  although  they  may  not  specifically 
address  risk,  deviation  from  these  policies 
represents  risk  to  a  program.  Internal  com¬ 
pany  reports  that  address  how  well  programs 
comply  with  policy  may  be  required  and  will 
provide  valuable  information. 

•  Risk  Monitoring  Report.  Contractors 
should  have  internal  tracking  metrics  and 
reports  for  each  moderate-  or  high-risk 
item.  These  metrics  may  be  used  to  deter¬ 
mine  the  status  of  risk  reduction  programs. 
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3 

RISK  MANAGEMENT  AND 
DOD  ACQUISITION  PROCESS 


3.1  INTRODUCTION 

This  Chapter  discusses  the  relationship  be¬ 
tween  risk  and  the  acquisition  process, 
describes  how  risk  is  considered  in  design 
of  the  Acquisition  Plan,  and  expresses  the 
need  to  consider  risk  as  early  in  the  pro¬ 
gram  as  possible.  Appendix  A  is  summary 
of  the  risk  management  requirements  that 
are  contained  in  DoDD  5000.1  and  DoD 
5000.2-R. 

3.2  OVERVIEW 

The  DoD  acquisition  process  for  the  manage¬ 
ment  of  programs  consists  of  a  series  of 
phases  designed  to  reduce  risk,  ensure 
affordability,  and  provide  adequate  informa¬ 
tion  for  decision  making.  Acquisition  offi¬ 
cials  are  encouraged  to  tailor  programs  to 
eliminate  phases  or  activities  that  result  in 
little  payoff  in  fielding  time  or  cost  savings. 
To  effectively  tailor  a  program,  one  needs  to 
understand  the  risks  present  in  the  program 
and  to  develop  a  plan  for  managing  these 
risks.  DoD  policy  calls  for  the  continual  as¬ 
sessment  of  program  risks,  beginning  with 
the  initial  phase  of  an  acquisition  program, 
and  the  development  of  management  ap¬ 
proaches  before  any  decision  is  made  to  en¬ 
ter  all  subsequent  phases. 

The  application  of  risk  management  pro¬ 
cesses  (planning,  assessment,  handling,  and 
monitoring)  is  particularly  important  during 
Phase  0  of  any  program,  when  alternatives 
are  evaluated,  program  objectives  are  estab¬ 
lished,  and  the  acquisition  strategy  is  devel¬ 


oped.  All  of  these  activities  require  accep¬ 
tance  of  some  level  of  risk  and  development 
of  plans  to  manage  the  risk. 

As  a  program  evolves  into  subsequent 
phases,  the  nature  of  the  risk  management 
effort  will  change  new  assessments  built 
on  previous  ones.  Risk  areas  will  become 
more  specific  as  the  system  is  defined. 

Risk  management  should  also  be  an  inte¬ 
gral  part  of  any  Source  Selection  process, 
from  RFP  preparation,  through  proposal 
evaluation,  and  after  contract  award. 
Throughout  the  program  life,  IPTs  will  play 
a  key  role  in  risk  management  activities. 

3.3  DOD  ACQUISITION  PROCESS 

The  phases  and  milestones  of  the  acquisition 
process  provide  a  streamlined  structure  that 
emphasizes  risk  management  and 
affordability.  The  phases  are  a  logical  means 
of  progressively  translating  broadly-stated 
mission  needs  into  weU-defined  system-spe¬ 
cific  requirements,  and  ultimately  into  op¬ 
erationally  effective,  suitable,  and  survivable 
systems.  It  is  important  to  remember  that 
the  term  "system"  includes  hardware,  soft¬ 
ware,  and  the  human  element.  Each  phase 
is  designed,  among  other  things,  to  manage 
risks.  Milestones  are  points  in  time  that  al¬ 
low  decision  makers  to  evaluate  the  program 
status  and  determine  if  the  program  should 
proceed  to  the  next  phase.  The  ^^estone  De¬ 
cision  Authority  (MDA)  and  PM  tailor  mile¬ 
stones  and  phases  so  that  each  milestone 
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decision  point  allows  assessment  of  program 
status  and  the  opportunity  to  review  plans 
for  the  next  phase  and  beyond.  The  MDA 
should  explicitly  address  program  risks  and 
the  adequacy  of  risk  management  planning 
during  the  iiulestone  reviews  and  establish 
exit  criteria  for  progression  to  the  next  phase. 

The  contract  schedule  should  allow  time 
for  milestone  decisions  before  spending  be¬ 
gins  in  subsequent  phases  and  should  also 
permit  demonstration  of  the  exit  criteria  in 
time  to  support  the  milestone  review.  The 
objective  is  to  provide  proper  fiscal  con¬ 
trol  without  delaying  the  acquisition  deci¬ 
sions  or  contracts  while  adequately  consid¬ 
ering  risk. 

The  acquisition  strategy  defines  the  business 
and  technical  management  approach  to  meet 
objectives  within  program  constraints  with 
a  primary  goal  to  minimize  the  time  and  cost 
of  satisfying  a  valid  need,  consistent  with 
common  sense  and  sound  business  practices. 
A  PM  prepares  a  preliminary  acquisition 
strategy  at  Milestone  0  (that  includes  Phase 
0  activities  that  focus  on  identifying  risk  and 
handling  options).  Later,  the  PM  updates  the 
strategy  to  support  each  milestone  decision 
by  describing  activities  and  events  planned 
for  the  upcoming  phase  and  relating  the  ac¬ 
complishments  of  that  phase  to  the  program's 
overall,  long-term  objectives.  The  risk  asso¬ 
ciated  with  a  program  will  significantly  in¬ 
fluence  the  acquisition  strategy. 

3.4  CHARACTERISTICS  OF  THE 
ACQUISITION  PROCESS 

The  acquisition  process  that  has  evolved 
can  be  characterized  in  terms  of  the  follow¬ 
ing  concepts  that  are  particularly  relevant 
to  the  management  of  risk  in  programs. 


3.4.1  Integrated  Product  and  Process  De¬ 
velopment  (IPPD) 

IPPD  integrates  all  acquisition  activities  in 
order  to  optimize  system  development, 
production,  and  deployment.  Key  to  the 
success  of  the  IPPD  concept  are  the  IPTs, 
which  are  composed  of  qualified  and  em¬ 
powered  representatives  from  all  appropri¬ 
ate  functional  disciplines  who  work  to¬ 
gether  to  identify  and  resolve  issues.  As 
such,  IPTs  are  the  foundation  for  organiz¬ 
ing  for  risk  management. 

3.4.2  Continuous  Risk  Management 

PMs  should  focus  on  risk  management 
throughout  the  life  of  the  program,  not  just 
in  preparation  for  program  and  milestone 
reviews.  Program  risks  should  be  continu¬ 
ously  assessed,  and  the  risk-handling  ap¬ 
proaches  developed,  executed,  and  moni¬ 
tored  throughout  the  acquisition  process. 
Both  the  Government  and  contractors  must 
understand  risks  as  a  program  progresses 
through  the  various  phases  and  milestone 
decision  points,  and  must  modify  the  man¬ 
agement  strategy  and  plan  accordingly. 

3.4.3  Program  Stability 

Once  a  program  is  initiated,  program  stabil¬ 
ity  is  a  top  priority.  Keys  to  creating  pro¬ 
gram  stability  are  realistic  investment  plan¬ 
ning  and  affordability  assessments.  They 
must  reflect  an  accurate  and  comprehensive 
imderstanding  of  existing  or  expected  pro¬ 
gram  risks.  A  risk  management  strategy 
must  be  developed  early  in  the  process,  be¬ 
fore  actually  initiating  the  program  to  ensure 
it  is  a  stable  one,  recognizing  that  key  issues 
affecting  program  stability  may  be  external. 
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3.4.4  Reduction  of  Life-Cycle  Costs 

DoD  considers  the  reduction  of  total  cost 
to  acquire  and  operate  systems  while  main¬ 
taining  a  high  level  of  performance  for  the 
user  to  be  of  highest  priority.  This  is  re¬ 
flected,  in  part,  through  the  introduction 
of  the  "Cost  As  an  Independent  Variable" 
(CAIV)  concept.  CAIV  entails  setting  ag¬ 
gressive,  realistic  cost  objectives  early  in  an 
acquisition  program  and  then  managing  all 
aspects  of  the  program  to  achieve  those  ob¬ 
jectives,  while  still  meeting  the  user's  per¬ 
formance  and  schedule  needs.  Inherent  in 
the  CAIV  concept  is  the  realization  that 
risks  must  be  understood,  taken,  and  man¬ 
aged  in  order  to  achieve  performance, 
schedule,  and  cost  objectives.  An  under¬ 
standing  of  risk  is  essential  to  setting  real¬ 
istic  cost  objectives.  The  PM  and  user  rep¬ 
resentatives  should  identify  risk  and  cost 
driving  requirements  during  the  genera¬ 
tion  of  the  Operational  Requirement  Docu¬ 
ment  (ORD)  in  order  to  know  where  trade¬ 
offs  may  be  necessary. 

3.4.5  Event-Oriented  Management 

Event-oriented  management  requires  that 
decision  makers  base  their  decisions  on  sig¬ 
nificant  events  in  the  acquisition  life  cycle, 
rather  than  on  arbitrary  calendar  dates. 
This  management  process  emphasizes  ef¬ 
fective  acquisition  planning  and  embodies 
sound  risk  management.  Decisions  to  pro¬ 
ceed  with  a  program  should  be  based  on 
demonstration  of  performance,  through 
test  and  evaluation,  and  on  verification  that 
program  risks  are  well-understood  and  are 
being  managed  effectively.  Attainment  of 
agreed-upon  exit  criteria  is  an  indication 
that  the  PMO  is  managing  risk  effectively. 

3.4.6  Modeling  and  Simulation 

Properly  used,  models  and  simulations  can 
reduce  time,  resources,  and  acquisition  risk 
and  may  increase  the  quality  of  the  systems 


being  developed.  Users  of  these  models 
and  simulations  must  have  a  good  under¬ 
standing  of  their  capabilities  and  limita¬ 
tions  and  their  applicability  to  the  issues 
being  addressed. 

From  a  risk  perspective,  modeling  and 
simulation  may  be  used  to  develop  alter¬ 
native  concepts  during  system  design;  pre¬ 
dict  performance  in  support  of  tradeoff 
studies;  evaluate  system  design  and  sup¬ 
port  preliminary  design  reviews  during 
design  development;  predict  system  per¬ 
formance  and  supplement  live  tests  dur¬ 
ing  testing;  examine  the  military  value  of 
the  system;  determine  the  impact  of  design 
changes;  hone  requirements;  and  develop 
hfe  cycle  support  requirements  and  assess¬ 
ments. 

3.5  RISK  MANAGEMENT 
ACTIVITIES  DURING 
ACQUISITION  PHASES 

Risk  management  activities  should  be  ap¬ 
plied  continuously  throughout  all  acquisi¬ 
tion  process  phases.  However,  because  of 
the  difference  in  available  information,  the 
level  of  application  and  detail  will  vary  for 
each  phase.  In  Phase  0,  management  fo¬ 
cuses  on  assessing  the  risks  in  the  alterna¬ 
tive  concepts  available  to  satisfy  users 
needs  and  on  planning  a  strategy  to  ad¬ 
dress  those  risks.  For  each  of  the  subse¬ 
quent  phases,  all  four  risk  management  ac¬ 
tivities  may  be  applied  with  increasing  fo¬ 
cus  on  risk  handling  and  monitoring. 

The  PM  identifies  objectives,  alternatives, 
and  constraints  at  the  beginning  of  each 
phase  of  a  program  and  then  evaluates  al¬ 
ternatives,  identifies  sources  of  project  risk, 
and  selects  a  strategy  for  resolving  the  risks. 
The  PMO  updates  the  acquisition  strategy, 
risk  assessments,  and  other  aspects  of  pro¬ 
gram  planning,  based  on  analyses,  for  the 
phase  of  the  acquisition. 
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Developers  should  become  involved  in  the 
risk  management  process  at  the  beginning, 
when  users  define  performance  require¬ 
ments,  and  continue  during  the  acquisition 
process  until  the  system  is  delivered.  The 
early  identification  and  assessment  of  criti¬ 
cal  risks  allow  PMs  to  formulate  handling 
approaches  and  to  streamline  the  program 
definition  and  the  RFP  around  critical 
product  and  process  risks. 

The  following  paragraphs  address  risk 
management  in  the  different  phases  in 
more  detail. 

3.5.1  Phase  0 

DoD  5000.2-R  describes  Phase  0  as  nor¬ 
mally  consisting  of  studies  that  define  and 
evaluate  the  feasibility  of  alternative  con¬ 
cepts  and  provide  the  basis  for  the  assess¬ 
ment  of  these  alternatives  in  terms  of  their 
advantages,  disadvantages,  and  risk  lev¬ 
els  at  the  Milestone  (MS)  I  decision  point. 
In  addition  to  the  Analysis  of  Alternatives, 
the  PM  develops  a  proposed  acquisition 
program  baseline  (APB)  and  exit  criteria 
for  Phase  I. 

The  APB  documents  the  most  important 
performance,  cost,  and  schedule  objectives 
and  thresholds  for  the  selected  concepts. 
The  parameters  selected  are  such  that  a  re- 
evaluation  of  alternative  concepts  is  appro¬ 
priate  if  thresholds  are  not  met.  Exit  crite¬ 
ria  are  events  or  accomplishments  that  al¬ 
low  managers  to  track  progress  in  critical 
technical,  cost,  or  schedule  risk  areas.  They 
must  be  demonstrated  to  show  that  a  pro¬ 
gram  is  on  track. 

In  defining  alternative  concepts,  PMs  should 
pay  particular  attention  to  the  threat  and  the 
user's  requirements,  which  are  normally 
stated  in  broad  terms  at  this  time.  Risks  can 
be  introduced  if  the  requirements  are  not 
stable,  or  if  they  are  overly  restrictive  and 


contain  specific  technical  solutions.  Require¬ 
ments  can  also  be  significant  cost  and  sched¬ 
ule  risk  drivers  if  they  require  a  level  of  per¬ 
formance  that  is  difficult  to  achieve  within 
the  program  budget  and  time  constraints. 
Such  drivers  need  to  be  identified  as  early  in 
the  program  as  possible. 

The  acquisition  strategy  should  address  the 
known  risks  for  each  alternative  concept, 
and  the  plans  to  handle  them,  including 
specific  events  intended  to  control  the  risks. 
Similarly,  the  test  and  evaluation  strategy 
should  reflect  how  T&E,  with  the  use  of 
modeling  and  simulation,  will  be  used  to 
assess  risk  levels  and  identify  new  or  sus¬ 
pected  risk  areas. 

A  risk  management  strategy,  derived  in  con¬ 
cert  with  the  acquisition  strategy,  should  be 
developed  during  this  phase  and  revised  and 
updated  continually  throughout  the  pro¬ 
gram.  This  strategy  should  include  risk  man¬ 
agement  planning  that  clearly  defines  roles, 
responsibilities,  authority,  and  documenta¬ 
tion  for  program  reviews,  risk  assessments, 
and  risk  monitoring. 

3.5.2  Subsequent  Phases 

Ehiring  subsequent  phases,  concepts,  tech¬ 
nological  approaches,  and/ or  design  ap¬ 
proaches  (selected  at  the  previous  mile¬ 
stone  decisions)  are  pursued  to  define  the 
program  and  program  risks.  Selected  al¬ 
ternative  concepts  continue  to  be  analyzed, 
and  the  acquisition  strategy,  and  the  vari¬ 
ous  strategies  and  plans  derived  from  it, 
continue  to  be  refined. 

Risk  management  efforts  in  these  phases  fo¬ 
cus  on:  imderstanding  critical  technology, 
manufacturing,  and  support  risks,  along  with 
performance,  cost,  and  schedule  risks;  and 
demonstrating  that  they  are  being  controlled 
before  moving  to  the  next  milestone.  Thus, 
particular  attention  should  be  placed  on  han- 
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dling  and  monitoring  activities.  Planning 
and  assessment  should  continue  as  new  in¬ 
formation  becomes  available  and  new  risk 
events  are  identified. 

During  these  phases,  the  risk  management 
program  should  be  carried  out  in  an  inte¬ 
grated  Government-contractor  framework 
that  allows  the  Government  to  manage  pro¬ 
gram  risks,  with  the  contractor  responsible 
to  the  PM  for  product  and  process  risks  and 
for  maintaining  design  accountability.  Both 
the  Government  and  contractors  need  to  im- 
derstand  the  risks  clearly,  and  jointly  plan 
management  efforts. 

3.6  RISK  MANAGEMENT  AND 
MILESTONE  DECISIONS 

Before  a  milestone  review,  the  PM  should  up¬ 
date  risk  assessments,  explicitly  addressing 
the  risks  in  the  critical  areas,  such  as  threat, 
requirements,  technology,  etc.,  and  identify 
areas  of  moderate  or  high  risk. 

Each  critical  assessment  should  be  sup¬ 
ported  by  subsystems'  risk  assessments, 
which  should  be  supported  by  design  re¬ 
views,  test  results,  and  specific  analyses. 

The  PM  should  present  planned  risk  miti¬ 
gation  actions  for  moderate  or  high  risk 
areas  at  the  milestone  review  to  determine 
their  adequacy  and  to  ensure  the  efficient 
allocation  of  resources. 

3.7  RISK  MANAGEMENT  AND  THE 
ACQUISITION  STRATEGY 

In  addition  to  providing  the  framework  for 
program  planning  and  execution,  the  ac¬ 
quisition  strategy  serves  several  purposes 
that  are  important  to  risk  management: 

•  Provides  a  master  schedule  for  re¬ 
search,  development,  test,  production,  de¬ 
ployment,  and  critical  events  in  the  acqui¬ 
sition  cycle. 


•  Gives  a  master  checklist  of  the  impor¬ 
tant  issues  and  alternatives  that  must  be 
addressed. 

•  Assists  in  prioritizing  and  integrating 
functional  requirements,  evaluating  alter¬ 
natives,  and  providing  a  coordinated  ap¬ 
proach  to  integrate  diverse  functional  is¬ 
sues,  leading  to  the  accomplishment  of  pro¬ 
gram  objectives. 

•  Documents  the  assumptions  and 
guidelines  that  led  to  the  initiation  and  di¬ 
rection  of  the  program. 

•  Provides  the  basis  for  the  development 
and  execution  of  the  various  subordinate 
functional  strategies  and  plans. 

The  strategy  structure  should  ensure  a 
sound  program  through  the  management 
of  performance,  schedule,  and  cost  risk.  A 
good  acquisition  strategy  acknowledges 
and  identifies  program  risks  and  forms  the 
basis  for  implementing  a  forward-looking, 
rather  than  reactive,  effective  risk  manage¬ 
ment  effort. 

Acquisition  strategy  should  describe  how 
risk  is  to  be  handled  and  identify  which 
risks  are  to  be  shared  with  the  contractor 
and  which  are  to  be  retained  by  Govern¬ 
ment.  The  key  concept  here  is  that  the  Gov¬ 
ernment  shares  the  risk  with  the  contrac¬ 
tor,  but  does  not  transfer  risk  to  the  con¬ 
tractor.  The  PMO  always  has  a  responsi¬ 
bility  to  the  system  user  to  develop  a  ca¬ 
pable  system  and  can  never  absolve  itself 
of  that  responsibility.  Therefore,  all  pro¬ 
gram  risks,  whether  primarily  managed  by 
the  PMO  or  by  the  contractor,  must  be  as¬ 
sessed  and  managed  by  the  PMO. 

Once  the  program  office  has  determined 
how  much  of  each  risk  is  to  be  shared  with 
the  contractor,  it  should  assess  the  total  risk 
assumed  by  the  developing  contractor  (in¬ 
cluding  subcontractors).  The  Government 
should  not  require  contractors  to  accept  fi- 
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nandal  risks  that  are  inconsistent  with  their 
ability  to  handle  them.  Financial  risks  are 
driven,  in  large  measure,  by  the  underly¬ 
ing  technical  and  programmatic  risks  in¬ 
herent  in  a  program.  The  Government  con¬ 
tracting  officer  should,  therefore,  select  the 
proper  type  of  contract  based  on  an  appro¬ 
priate  risk  assessment,  to  ensure  a  clear  re¬ 
lationship  between  the  selected  contract 
t)q?e  and  program  risk. 

3.8  RISK  MANAGEMENT  AND  CAIV 

The  intention  of  CAIV  is  to  establish  balance 
between  cost,  schedule,  and  performance 
early  in  the  acquisition  process  and  to  man¬ 
age  to  the  cost  objective.  CATV  requires  that 
PMs  establish  aggressive  cost  objectives,  de¬ 
fined  to  some  degree  by  the  maximum  level 
of  acceptable  risL  Risks  in  achieving  both 
performance  and  aggressive  cost  goals  must 
be  clearly  recognized  and  actively  managed 
through 

(1 )  continuing  iteration  of  cost/ perfor¬ 
mance/schedule/risk  tradeoffs, 

(2)  identifying  key  performance  and 
manufacturing  process  uncertainties, 

(3)  and  demonstrating  solutions  before 
production. 

Whereas  DoD  has  traditionally  managed 
performance  risk,  equal  emphasis  must  be 
placed  on  managing  cost  and  supportabil- 
ity  goals.  An  underlying  premise  of  CAIV 
is  that  if  costs  are  too  great,  and  there  are 
ways  to  reduce  them,  then  the  user  and 


developer  may  reduce  performance  re¬ 
quirements  to  meet  cost  objections.  Cost 
control  and  effective  risk  management  in¬ 
volve  planning  and  scheduling  events  and 
demonstrations  to  verify  solutions  to  cost/ 
risk  problems. 

User  participation  in  the  tradeoff  analysis  is 
essential  to  attain  a  favorable  balance  be¬ 
tween  performance,  cost,  schedule,  and  risk. 
The  PM  and  user  representatives  should 
identify  risk  and  cost  driving  requirements 
during  the  generation  of  the  ORD  to  know 
where  tradeoffe  may  be  necessary.  Risk  as¬ 
sessments  are  critical  to  the  CAIV  process 
since  they  provide  users  and  developers  with 
essential  data  to  assist  in  the  cost-perfor¬ 
mance  trade  decisions. 

A  cost  for  risk  management  is  directly  re¬ 
lated  to  the  level  of  risk  and  affects  a  pro¬ 
gram  in  two  ways.  First,  costs  are  associ¬ 
ated  with  specific  handling  activities,  for 
example,  a  parallel  development.  Second, 
funds  are  needed  to  cover  the  known  risks 
of  the  selected  system  approach  (i.e.,  funds 
to  cover  cost  uncertainty).  PMs  must  in¬ 
clude  the  anticipated  expense  of  manag¬ 
ing  risk  in  their  estimates  of  program  costs. 
Decision  makers  must  weigh  these  costs 
against  the  level  of  risk  in  reaching  pro¬ 
gram  funding  decisions.  CAIV  requires 
that  program  funds  support  the  level  of 
accepted  program  risk  and  that  risk  man¬ 
agement  costs  are  included  in  setting  cost 
objectives. 
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4 

RISK  MANAGEMENT  AND  PROGRAM 
MANAGEMENT 


4.1  INTRODUCTION 

Risk  management  as  a  program  manage¬ 
ment  responsibility  can  be  a  comprehen¬ 
sive  and  responsive  management  tool  if  it 
is  properly  organized  and  monitored  at  the 
PM  level.  A  formalized  risk  management 
program  should  be  forward-looking  by 
identifying,  analyzing,  and  resolving  po¬ 
tential  problem  areas  before  they  occur,  and 
by  incorporating  monitoring  techniques 
that  accurately  portray  the  status  of  risks 
and  the  efforts  to  mitigate  them.  Introduc¬ 
tion  of  risk  management  early  in  a  program 
emphasizes  its  importance  and  encourages 
contractors  and  members  of  the  Govern¬ 
ment  team  to  consider  risk  in  the  daily 
management  functions. 

This  chapter  addresses  the  relationship  be¬ 
tween  risk  management  and  program  man¬ 
agement  and  suggests  methods  of  intro¬ 
ducing  risk  management  in  a  program,  or¬ 
ganizing  for  risk,  and  training. 

4.2  OVERVIEW 

A  PMO  should  organize  for  risk  manage¬ 
ment,  using  existing  IPTs.  The  PM  may 
also  want  to  use  contractors  to  support 
management  efforts  or  have  experts  not  in¬ 
volved  with  the  program  perform  indepen¬ 
dent  assessments. 

To  use  risk  management  as  a  program  man¬ 
agement  tool,  the  information  resulting 
from  each  of  the  risk  processes  should  be 
documented  in  a  usable  form  and  available 


to  members  of  the  Government/industry 
program  team.  This  information  will  pro¬ 
vide  the  basis  for  reporting  risk  and  over¬ 
all  program  information,  both  internally 
and  externally.  Managing  collection  and 
dissemination  of  risk  information  can  be 
enhanced  through  the  use  of  an  MIS. 

4.3  PROGRAM  MANAGER  AND 
RISK  MANAGEMENT 

All  PMs  are  responsible  for  establishing  and 
executing  a  risk  management  program  that 
satisfies  the  policies  contained  in  DoDD 
5000.1.  A  PM  must  balance  program-unique 
requirements  or  circumstances  (e.g.,  size  of 
the  PMO  staff)  against  the  demands  of 
proven  risk  management  principles  and 
practices.  This  section  addresses  these  prin¬ 
ciples  and  practices  and  provides  a  basis  for 
estabhshing  a  PMC's  risk  management  or¬ 
ganization  and  related  procedures.  The  fol¬ 
lowing  are  guidelines  for  defining  an  ap¬ 
proach  to  risk  management. 

4.3.1  Risk  Management  Is  a 

Program  Management  Tool 

Risk  management  should  be  integral  to  a 
program's  overall  management.  PMs  must 
take  an  active  role  in  the  process  to  ensure 
that  their  approach  leads  to  a  balanced  use 
of  program  resources,  reflects  their  overall 
management  philosophy,  and  includes 
Government  and  contractors.  Past  DoD 
practices  have  generally  treated  risk  man¬ 
agement  as  a  system  engineering,  cost-es¬ 
timating  technique  or  possibly  as  an  inde- 
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pendent  function  distinct  from  other  pro¬ 
gram  functions.  Today,  risk  management 
is  recognized  as  a  vital  integrated  program 
management  tool  that  cuts  across  the  en¬ 
tire  acquisition  program,  addressing  and 
interrelating  cost,  schedule,  and  perfor¬ 
mance  risks.  The  goal  is  to  make  everyone 
involved  in  a  program  aware  that  risk 
should  be  a  consideration  in  the  design,  de¬ 
velopment,  and  fielding  of  a  system.  It 
should  not  be  treated  as  someone  else's  re¬ 
sponsibility. 

4.3.2  Risk  Management  Is  a 
Formal  Process 

Formal  risk  management  refers  to  a  struc¬ 
tured  process  whereby  risks  are  systemati¬ 
cally  identified,  analyzed,  and  controlled. 
(A  recommended  structure  is  described  in 
Section  2  of  this  guide.)  A  structured  risk 
management  process,  which  is  applied 
early,  continuously,  and  rigorously,  pro¬ 
vides  a  disciplined  environment  for  deci¬ 
sion  making  and  for  the  efficient  use  of 
program  resources.  Through  a  disciplined 
process  PMs  can  uncover  obscure  and 
lower-level  risks  that  collectively  could 
pose  a  major  risk. 

The  need  for  a  formal  risk  management 
process  arises  from  the  nature  of  risk  and 
the  complexity  of  acquisition  programs. 
The  numerous  risks  in  an  acquisition  pro¬ 
gram  are  often  interrelated  and  obscure 
and  change  in  the  course  of  the  develop¬ 
ment  process.  A  formal  approach  is  the 
only  effective  method  to  sort  through  nu¬ 
merous  risk  events,  to  identify  the  risks  and 
their  interrelationships,  to  pinpoint  the 
truly  critical  ones,  and  to  identify  cost-ef¬ 
fective  ways  to  reduce  those  risks,  consis¬ 
tent  with  overall  program  objectives. 

A  structured  process  can  reduce  the  complex¬ 
ity  of  an  acquisition  program  by  defining  an 
approach  to  assess,  handle,  and  communi¬ 


cate  program  risk.  The  systematic  identifi¬ 
cation,  analysis,  and  mitigation  of  risks  also 
offers  a  reliable  way  to  ensure  objectivity,  that 
is,  minimize  unwarranted  optimism,  preju¬ 
dice,  ignorance,  or  self-interest.  Further, 
structure  reduces  the  impact  of  persoimel 
turnover  and  provides  a  basis  for  training 
and  consistency  among  aU  the  functional  ar¬ 
eas  of  a  program.  A  structured  risk  program 
may  also  promote  teamwork  and  under¬ 
standing  and  improves  the  qualify  of  the  risk 
products. 

4.3.3  Risk  Management  Is 
Forward-Looking 

Effective  risk  management  is  based  on  the 
premise  that  PMs  must  identify  potential 
problems,  referred  to  as  risk  events,  long 
before  they  can  occur  and  develop  strate¬ 
gies  that  increase  the  likelihood  of  a  favor¬ 
able  outcome  to  these  problems.  Applica¬ 
tion  of  this  philosophy  occurs  primarily  by 
using  analytical  techniques  that  give  for¬ 
ward-looking  assessments. 

Typically,  the  early  identification  of  poten¬ 
tial  problems  is  concerned  with  two  types 
of  events.  The  first  are  relevant  to  the  cur¬ 
rent  or  imminent  acquisition  phase  of  a 
program  (intermediate-term),  such  as  sat¬ 
isfying  a  technical  exit  criteria  in  time  for 
the  next  milestone  review.  The  second  are 
concerned  with  the  future  phase(s)  of  a 
program  (long-term)  such  as  potential  risk 
events  related  to  transitioning  a  system 
from  development  to  production. 

By  analyzing  critical  events,  certain  risks 
can  be  determined.  To  do  this,  one  should 
consider  the  range  of  potential  outcomes 
and  the  factors  that  determine  those  out¬ 
comes.  Through  risk  handling,  a  PM  then 
develops  approaches  that  minimize  risk 
factors.  Paragraph  5.6  of  this  Guide  de¬ 
scribes  some  handling  approaches. 
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Choosing  the  proper  risk-handling  options 
requires  that  a  balance  be  struck  between 
the  resources  required  to  implement  those 
options  (both  intermediate  and  long-term) 
and  the  resources  realistically  available. 

4.3.4  Risk  Management  Is  Integral  to 
Integrated  Product  and  Process 
Development  (IPPD) 

One  of  the  tenets  of  IPPD  is  multidisci¬ 
plinary  teamwork  through  IPTs,  which  are 
an  integral  part  of  the  defense  acquisition 
oversight  and  review  process.  The  Inte¬ 
grating  IPT  (IIPT)  is  a  valuable  resource  to 
assist  in  developing  a  risk  management 
plan  and  should  be  used  accordingly.  The 
PM  should  ensure  that  the  requirements  of 
the  overarching  IPT  (OIPT)  are  reflected  in 
the  plan. 

Working  with  the  OIPT,  the  PM  can  estab¬ 
lish  the  t5q)e  and  frequency  of  risk  manage¬ 
ment  information  that  an  OIPT  requires,  and 
refine  management  organization  and  proce- 
dmes.  This  should  be  done  during  the  ini¬ 
tial  OIPT  meetings.  OIPTs  will  most  likely 
require  information  concerning: 

•  Known  risks  and  their  characteristics, 
e.g.,  probability  of  occurrence  and  conse¬ 
quences, 

•  Planned  risk-handling  actions,  funded 
and  unfunded, 

•  Achievements  in  controlling  risks  at 
acceptable  levels. 

IIPTs  and  OIPTs  may  also  require  details 
on  the  PM's  risk  management  program,  ac¬ 
cess  to  the  risk  management  plan,  and  the 
results  of  specific  risk  assessments.  In  ad¬ 
dition,  PMs  may  want  to  present  selected 
information  to  IIPTs  and  OIPTs  to  help  sub¬ 
stantiate  a  position  or  recommendation, 
e.g.,  help  support  a  budget  request. 


4.4  RISK  MANAGEMENT 

ORGANIZATION  IN  THE  PMO 

The  PM,  after  determining  a  preferred  man¬ 
agement  approach,  must  organize  the  pro¬ 
gram  office  and  establish  outside  relation¬ 
ships  in  order  to  manage  risk.  No  particular 
organizational  structure  is  superior;  how¬ 
ever,  experience  provides  some  insights  into 
the  development  of  effective  risk  manage¬ 
ment  organizations.  PMs  should  consider 
the  following  discussion  in  the  context  of 
their  unique  requirements  and  circumstances 
and  apply  those  that  are  suitable  to  their  spe¬ 
cific  needs. 

4.4.1  Risk  Management  Organizational 
Structure 

A  major  choice  for  each  PM  is  whether  to 
have  a  centralized  or  decentralized  risk  man¬ 
agement  organization.  The  PM  may  choose 
a  centralized  organizational  structure  xmtil 
team  members  become  familiar  witti  both  the 
program  and  the  risk  management  process. 
Then  the  PM  may  choose  to  decentralize.  The 
degree  of  decentralization  depends  on  the 
assignment  of  responsibilities. 

In  a  centralized  approach,  the  PM  estab¬ 
lishes  a  team  that  is  responsible  for  all  as¬ 
pects  of  risk  management.  The  team  would 
write  a  plan,  conduct  assessments,  evalu¬ 
ate  risk-handling  options,  and  monitor 
progress.  Although  this  approach  may  be 
necessary  early  in  a  program,  it  tends  to 
minimize  the  concept  that  risk  manage¬ 
ment  is  a  responsibility  shared  by  all  mem¬ 
bers  of  the  acquisition  team,  whether  Gov¬ 
ernment  or  contractor. 

The  suggested  approach  is  a  decentralized 
risk  management  organization,  because  it 
is  compatible  with  the  DoD's  IPPD  policy 
and  generally  results  in  an  efficient  use  of 
personnel  resources.  The  following  guide¬ 
lines  apply  to  all  risk  management  organi¬ 
zations: 
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•  The  PM  is  ultimately  responsible  for 
planning,  allocating  resources,  and  execut¬ 
ing  risk  management.  This  requires  the  PM 
to  oversee  and  participate  in  the  risk  man¬ 
agement  process. 

•  The  PM  must  make  optimal  use  of  avail¬ 
able  resources,  i.e.,  personnel,  organizations, 
and  funds.  Personnel  and  organizational  re¬ 
sources  include  the  PMO,  functional  support 
offices  of  the  host  command,  the  prime  con¬ 
tractor,  independent  risk  assessors,  and  sup¬ 
port  contractors. 

•  Risk  management  is  a  team  function. 
This  stems  from  the  pervasive  nature  of  risk 
and  the  impact  that  risk-handling  plans 
may  have  on  other  program  plans  and  ac¬ 
tions.  In  the  aggregate,  risk  assessment, 
risk  handling,  and  risk  monitoring  affect 
all  program  activities  and  organizations. 
Any  attempt  to  implement  an  aggressive 
forward-looking  risk  management  pro¬ 
gram  without  the  involvement  of  all  PMO 
subordinate  organizations  could  result  in 
confusion,  misdirection,  and  wasted  re¬ 
sources.  The  only  way  to  avoid  this  is 
through  teamwork  among  the  PMO  orga¬ 


nizations  and  the  prime  contractor.  The 
management  organizational  structure  can 
promote  teamwork  by  requiring  strong 
connectivity  between  that  structure,  the 
various  PMO  organizations,  and  the  prime 
contractor.  The  teams  may  use  indepen¬ 
dent  assessments  to  assist  them,  when  re¬ 
quired. 

Figure  4-1  portrays  a  decentralized  risk  man¬ 
agement  organization.  This  example  includes 
the  entire  PMO  and  select  non-PMO  organi¬ 
zations,  e.g.,  the  prime  contractor,  who  are 
members  of  the  IPTs.  The  figure  shows  that 
risk  management  is  an  integral  part  of  pro¬ 
gram  management  and  not  an  additional  or 
separate  function  to  perform.  Hence,  sepa¬ 
rate  persoimel  are  not  designated  to  manage 
risk,  but  rather  aU  individuals  are  required 
to  consider  risk  management  as  a  routine  part 
of  their  jobs.  As  shown,  this  organizational 
structure  is  suited  to  ACAT 1  programs,  but 
PMs  can  tailor  it  to  satisfy  their  specific  re¬ 
quirements. 

The  organizational  structure  shows  that  the 
PM  is  ultimately  responsible  for  risk  man- 
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Figure  4-1.  Decentralized  Risk  Management  Organization 
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agement.  There  is  a  coordinator  to  assist 
with  this  responsibility  and  act  as  an  "op¬ 
erations"  officer.  This  may  be  a  full-time 
position  or  an  additional  duty  as  the  PM 
deems  appropriate.  The  coordinator 
should  have  specific  training  and  experi¬ 
ence  in  risk  management  to  increase  the 
chance  of  successful  implementation  and 
to  avoid  common  problems.  A  support 
contractor  may  assist  the  coordinator  by 
performing  administrative  tasks  associated 
with  that  office. 

The  Program  Level  IPT,  composed  of  indi¬ 
viduals  from  the  PMO  and  prime  contrac¬ 
tor,  ensures  that  the  PM's  risk-management 
program  is  implemented  and  that  the  pro¬ 
gram  results  are  synthesized  into  a  form 
suitable  for  decision  making  by  the  PM  and 
OIPT. 

The  inclusion  of  both  Sub-Tier  IPTs  and 
PMO  functional  offices  simply  reflects  that 
not  all  program  management  functions  will 
be  assigned  to  Sub-Tier  IPTs  for  execution. 

Independent  risk  assessors  are  typically 
hired  when  the  PM  has  specific  technical 
concerns  with  a  hardware  or  software 
product  or  engineering  process  and  wants 
an  independent  assessment  from  an  expert 
in  a  particular  field.  The  duration  of  their 
services  is  normally  short,  and  their  results 
are  reported  directly  to  the  PM. 

4.4.2  Risk  Management 
Responsibilities 

This  section  identifies  the  primary  respon¬ 
sibilities  that  could  be  associated  with  a 
decentralized  risk  management  organiza¬ 
tion.  In  assigning  the  responsibilities  to  the 
various  organizational  elements,  the  PM 
should  strike  a  balance  between  a  concen¬ 
tration  of  responsibilities  at  the  higher  lev¬ 
els  and  pushing  them  too  far  down  the  or¬ 
ganizational  structure. 


The  development  of  these  responsibilities, 
in  part,  is  based  on  the  premise  that  risk 
management  activities  must  be  specific — 
and  assigned  to  individuals,  not  groups. 
The  responsibilities  listed  below  are  as¬ 
signed  to  the  leader  of  each  organizational 
element,  recognizing  that  the  composition 
of  each  element  will  be  program  unique, 
i.e.,  number  of  assigned  PMO  personnel, 
prime  contractor  personnel,  etc.  The  task 
of  further  assigning  these  responsibilities, 
along  with  tailoring  them  to  satisfy  the 
needs  and  requirements  of  each  program, 
remains  for  PMs  and  their  staffs  to  accom¬ 
plish. 

Table  4-1  provides  a  description  of  the  re¬ 
sponsibilities  associated  with  the  decentral¬ 
ized  risk  management  structure,  sorted  by 
notional  organizational  elements  that  may 
make  up  the  risk  management  structure. 

4.5  CONTRACTOR  RISK 
MANAGEMENT 

Experience  has  shown  that  managing  a 
program's  risks  requires  a  close  partner¬ 
ship  between  the  PMO  and  the  prime 
contractor(s).  PMs  must  determine  the 
type  of  support  they  need  from  their  prime 
contractor,  communicate  these  needs 
through  the  Request  for  Proposal  (RFP)  for 
each  acquisition  phase,  and  then  provide 
for  them  in  the  contract.  Preparation  of  the 
RFP  and  source  selection  are  discussed  in 
subsequent  sections. 

4.5.1  Contractor  View  of  Risk 

Contractors  treat  risk  differently  from  the 
Government  because  each  views  risk  from 
a  different  perspective.  To  determine  how 
best  to  use  contractors,  the  PM,  in  execut¬ 
ing  his  risk  management  program,  needs 
to  understand  the  contractor  viewpoint. 

Contractors  t5q?ically  divide  risks  into  two 
basic  types:  business  risks  and  program  risks. 
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Table  4-1.  Notional  Description  of  Risk  Management  Responsibiiities 


Personnel 

Job  Responsibility 

Program  Manager 

Plan,  organize,  direct,  and  control  risk  management. 

Comply  with  DoDD  5000.1,  DoD  5000.2-R,  DoDD  5000.4,  and  DoD 
5000.4-M  risk  management  requirements. 

Ensure  that  funds  are  available  to  support  approved  risk-handling  plans. 

Inform  and  advise  MDA  Cost  Analysis  Improvement  Group  (CAIG)  and 
OIPT  on  program  risk  and  its  mitigation. 

Risk  Management 
Coordinator 

Develop  and  maintain  risk  management  plans. 

Provide  risk  management  training. 

Define  the  risk  reporting  scales  to  be  used  by  the  program. 

Develop  and  maintain  a  risk  management  information  system. 

Prepare  risk  management  reports. 

Monitor  compliance  with  DoDD  risk  management  requirements. 

Ensure  that  risk  management  functions  and  tasks  performed  by  the  Sub- 
Tier  IPTs  and  the  PMO  functional  offices  are  fully  integrated  and  in 
compliance  with  assigned  tasks. 

Advise  the  PM  and  Program  Level  IPT  on  the  use  of  risk  management 
sources,  i.e.,  host  command  functional  support  offices,  etc. 

Evaluate  risk  assessments,  risk-handling  plans,  and  risk  monitoring  results 
as  directed  and  recommend  appropriate  actions. 

Advise  the  PM  on  the  use  of  independent  risk  assessors. 

Program  Levei  IPT 

Ensure  that  the  risk  management  program  is  implemented,  risk  reduction 
is  accomplished  in  conformance  with  the  PM’s  strategy,  and  the  risk 
management  efforts  of  the  Sub-Tier  IPTs  are  integrated. 

Report  risk  events  to  the  risk  management  coordinator. 

Evaluate  whether  Sub-Tier  IPTs  and  PMO  functional  offices  have 
identified  critical  risks  and  proposed  risk-handling  plans. 

Ensure  that  cost,  schedule,  and  performance  risks  are  compatible. 

Ensure  that  cost,  schedule,  and  performance  risks  are  combined  in  a 
manner  consistent  with  the  plan. 

PMO  Sub-Tier  IPTs  & 
Functional  Offices 
(Process)  and  System 
Elements  (Products) 

Assess  risks,  recommending  appropriate  risk-handling  strategies  for  each 
identified  moderate  and  high  risk,  developing  and  implementing  risk¬ 
handling  plans,  monitoring  the  results  of  risk  mitigation  actions,  and 
documenting  all  risk  management  analyses  and  findings  within  the  team’s 
product  area. 

Coordinate  all  risk  management  findings  and  decision  with  other  Sub-Tier 
IPTs,  PMO  functional  offices,  the  Program  Level  IPT,  and  the  risk- 
management  coordination  office. 

Identify  funding  requirements  to  implement  risk-handling  plans. 

Identify  the  need  for  risk  management  training. 

Report  risk  events  to  the  risk  coordinator. 

Independent  Risk 
Assessors 

Perform  risk  assessment  on  critical  risk  areas  or  contractor  engineering 
processes  that  the  PM  has  specified. 

Report  the  results  of  those  assessments  to  the  PM. 

Work  with  the  risk  management  coordinator. 
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Business,  in  the  broadest  sense,  involves  the 
inherent  chance  of  making  a  profit  or  incur¬ 
ring  a  loss  on  any  given  contract.  Program 
risk  involves,  among  other  things,  technical, 
requirement,  and  design  uncertainties.  A 
contractor's  efforts  to  minimize  business  risks 
may  conflict  with  a  Government  PM's  efforts 
to  lower  program  risk. 

A  contractor's  view  of  program  risk  is  simi¬ 
lar  to  that  of  the  Government  except  for 
business  requirements  that  corporate  man¬ 
agement  has  placed  on  the  project.  The 
similarity,  however,  does  not  necessarily 
lead  to  the  contractor  having  a  competent 
internal  risk  management  program.  As  a 
Project  Management  Institute  (PMI)  hand¬ 
book  points  out,  "On  most  (contractor) 
projects,  responsibility  for  Project  Risk  is  so 
pervasive  that  it  is  rarely  given  sufficient 
central  attention."  As  a  minimum,  it  is 
important  that  the  PMO  writes  the  RFP 
asking  the  contractor  to  describe  his  risk 
management  process,  including  his  ap¬ 
proach  to  managing  any  specific  areas. 

4.5.2  Govemment/Contractor 
Relationship 

The  prime  contractor's  support  and  assis¬ 
tance  is  required  even  though  the  ultimate 
responsibility  for  risk  management  rests 
with  the  Government  PM.  Often,  the  con¬ 
tractor  is  better  equipped  to  understand  the 
program  technical  risks  than  the  Govern¬ 
ment  program  office  is.  Both  the  Govern¬ 
ment  and  contractor  need  to  share  infor¬ 
mation,  understand  the  risks,  and  develop 
and  execute  management  efforts.  The  Gov¬ 
ernment  must  involve  the  contractor  early 
in  program  development,  so  that  effective 
risk  assessment  and  reduction  can  occur. 

Therefore,  risk  management  must  be  a  key 
part  of  the  contractor's  management  scheme. 
Although  the  Government  does  not  dictate 
how  the  contractor  should  manage  risk,  some 


characteristics  of  a  good  Govemment/con- 
tractor  relationship  include: 

•  Clear  definition  of  risks  and  their  as¬ 
signment. 

•  Flexibility  for  assignment  of  risks  and 
risk  management  responsibilities  among 
the  teams. 

•  Strong  emphasis  on  best  management 
and  technical  practices  which,  if  followed, 
avoid  unnecessary  risks. 

Regarding  RFP  development,  discussed 
later  in  this  section,  information  is  pro¬ 
vided  on  how  these  characteristics  should 
be  addressed. 

The  Government/contractor  partnership 
can  be  forged  in  at  least  two  ways.  First, 
the  PMO  should  include  the  prime 
contractor(s)  in  the  top-level  risk  planning 
and  assessment  activities.  This  includes 
understanding  and  factoring  in  such  issues 
as  user  requirements,  affordability  con¬ 
straints,  and  schedule  limitations.  Second, 
the  PMO  should  include  in  advance  spe¬ 
cific  risk  assessment  and  handling  tasks  as 
key  contractual  efforts  during  the  concept 
exploration  and  program  definition  and 
risk  reduction  phases. 

Forming  a  joint  Government/contractor 
evaluation  team  is  a  good  way  of  fostering 
an  effective  partnership.  This  is  especially 
true  in  a  program's  early  stages  when  uncer¬ 
tainty  is  high  and  both  parties  must  fre¬ 
quently  assess  risks.  These  assessments, 
properly  handled,  involve  multidisciplinary 
efforts  requiring  subject  matter  experts  from 
both  the  prime  contractor  and  Government. 
This  joint  team  should  evaluate  the  proposed 
program  in  detail  and  explore  the  inherent 
program  risks,  the  proposed  handling  strat¬ 
egies,  the  detailed  development  schedule, 
and  the  contractor's  developmental  resources 
(people,  facilities,  processes,  tools,  etc.). 
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A  management  approach  using  multiple 
teams  is  the  best  approach  to  use,  e.g.,  Sub- 
Tier  IPTs.  Joint  team(s)  should  be  estab¬ 
lished  at  the  beginning  of  each  develop¬ 
ment  phase  to  assess  the  risks  to  be  over¬ 
come  in  that  phase  and  to  determine  the 
handling  technique(s)  to  be  used.  Require¬ 
ments  for  contractor  participation  on  the 
team(s)  should  be  identified  in  the  RFP  and 
subsequent  contract. 

4.6  RISK  MANAGEMENT  AND  THE 
CONTRACTUAL  PROCESS 

4.6.1  Risk  Management: 

Pre-Contract  Award 

The  contractor's  developmental  and  manu¬ 
facturing  processes  and  tools,  the  availabil¬ 
ity  and  skill  of  personnel,  and  the  previ¬ 
ous  experience  of  the  Government  and  con¬ 
tractor  team  all  influence  their  ability  to 
handle  the  proposed  system  development 
and  production.  Therefore,  an  effective  risk 
management  process  includes  an  evalua¬ 
tion  of  the  capabilities  of  the  potential  con¬ 
tractors. 

4.6.2  Early  Industry  Involvement: 

Industrial  Capabilities  Review 

An  Industrial  Capabilities  Review  is  a  pow¬ 
erful  tool  available  to  PMs  for  determin¬ 
ing  general  industrial  capabilities.  To  avoid 
potential  problems  in  the  subsequent  com¬ 
petitive  process  and  to  ensure  that  a  "level 
playing  field"  is  maintained,  an  announce¬ 
ment  in  the  Commerce  Business  Daily 
should  be  made  to  inform  all  potential 
offerors  that  the  Government  plans  to  con¬ 
duct  an  Industrial  Capabilities  Review  and 
to  request  responses  from  all  interested 
parties.  Below  is  a  general  approach  that 
PMOs  may  find  readily  adaptable  to  any 
type  of  capability  review.  The  basic  steps 
in  the  process  are  to: 

•  Obtain  the  Source  Selection  Authority's 
approval  to  conduct  the  review. 


•  Establish  the  criteria  for  the  capability. 

•  Identify  the  potential  contractors  who 
will  participate  in  the  review. 

•  Provide  an  advanced  copy  of  the  review 

material  to  those  contractors. 

•  Select  the  review  team,  ensuring  that 
it  has  the  necessary  mix  of  talent. 

•  Train  the  team  on  the  purpose  of  the 
review  and  review  criteria. 

•  Conduct  the  review  and  evaluate  the 
results. 

•  Provide  feedback  to  each  contractor  on 
the  results  of  their  review  and  assessment. 

•  Provide  the  results  to  the  PM. 

This  review  is  an  appraisal  of  general  in¬ 
dustrial  capabilities  and  supports  identi¬ 
fying  potential  program  risks  and  best 
practices  rather  than  evaluating  specific 
contractors. 

Regardless  of  the  approach,  the  PMO 
should  determine  what  specific  informa¬ 
tion  is  needed.  DoD  4245.7-M  is  a  good 
guide  to  help  tailor  a  set  of  questions  for 
the  contractors.  The  questions  generally 
focus  on  two  areas: 

•  What  is  the  state-of-the-art  of  the  tech¬ 
nology  proposed  for  use  in  the  system? 

•  What  are  the  general  developmental/ 
manufacturing  capabilities  of  the  potential 
contractors  (including  experience,  tools, 
processes,  etc.)  as  compared  to  industry 
best  practices? 

Table  4-2  shows  some  of  the  specific  areas  or 
sources  for  risk  identification.  It  includes  a 
number  of  areas  (threat,  requirements,  de¬ 
sign,  etc.)  that  have  been  shown  through  ex¬ 
perience  to  contain  risk  events  that  tend  to 
be  more  critical  than  others,  and  which  ones 
should  receive  the  most  management  atten¬ 
tion.  Risk  events  are  determined  by  examin- 
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ing  WBS  element  product  and  processes  in 
terms  of  risk  areas.  Process  areas  are  specifi¬ 
cally  addressed  in  DoD  4245.7M.  They  are 
general  in  that  areas  of  risk  could  be  present 
in  any  program  from  either  source  (WBS  or 
process).  They  are  intended  as  a  list  of  "top- 
level"  risk  sources  that  will  focus  attention 
on  a  specific  area.  The  PMO  and  contractor(s) 
win  have  to  examine  lower  levels  to  under¬ 
stand  the  actual  risks  that  are  present  in  their 
program  and  to  develop  an  effective  man¬ 
agement  plan.  The  risks  shown  are  not  in¬ 
tended  to  serve  as  a  simple  checklist  that  one 
should  apply  directly,  then  consider  the  pro¬ 
gram  risk-free  if  none  of  the  listed  risks  are 
present. 

An  examination  of  the  program  in  these  ar¬ 
eas  can  help  to  develop  the  final  program 
acquisition  strategy  and  the  risk-sharing 
structure  between  the  Government  and  in¬ 
dustry.  The  PMO  can  also  use  the  results 
to  adjust  the  RFP  for  the  next  phase  of  the 
program. 

4.6.3  Developing  the  Request  for 
Proposal 

The  RFP  should  communicate  to  all 
offerors  the  concept  that  risk  management 
is  an  essential  part  of  the  Government's 
acquisition  strategy. 

Before  the  draft  RFP  is  developed  using  the 
results  of  the  Industrial  Capabilities  Re¬ 
view,  the  PMO  should  conduct  a  risk  as¬ 
sessment  to  ensure  that  the  program  de¬ 
scribed  in  the  RFP  is  executable  within  the 
technical,  schedule,  and  budget  con¬ 
straints.  Based  on  this  assessment,  a  pro¬ 
gram  plan  and  integrated  master  schedule 
and  a  life-cycle  cost  (LCC)  estimate  may 
be  prepared.  The  technical,  schedule,  and 
cost  issues  should  be  discussed  in  the  pre¬ 
proposal  conference(s)  before  the  draft  RFP 
is  released.  In  this  way,  critical  risks  in¬ 
herent  in  the  program  can  be  identified  and 


addressed  in  the  RFP.  In  addition,  this 
helps  to  establish  key  risk-management 
contractual  conditions.  The  RFP  should 
encourage  offerors  to  extend  the  contract 
WBS  (CWBS)  to  reflect  how  they  will  iden¬ 
tify  all  elements  at  any  level  that  are  ex¬ 
pected  to  be  high  cost  or  high  risk.  The 
RFP  should  also  encourage  offerors  to  cite 
any  elements  of  the  CWBS  provided  in  the 
draft  RFP  that  are  not  consistent  with  their 
planned  approach. 

In  the  solicitation,  PMs  may  ask  offerors  to 
include  a  risk  analysis  and  a  description  of 
their  management  plans,  and  also  to  develop 
a  supporting  program  plan  and  an  integrated 
master  schedule  in  their  proposals.  These 
proposals  will  support  the  Government's 
source  selection  evaluation  and  the  formula¬ 
tion  of  a  most  probable  cost  estimate  for  each 
proposal.  In  addition,  the  RFP  may  identify 
the  requirement  for  periodic  risk-assessment 
reports  that  would  serve  as  inputs  to  the  PM's 
assessment  and  monitoring  processes 
thereby  ensuring  that  risks  are  continuously 
assessed. 

4.6.4  The  Offeror's  Proposal 

The  offerors  should  develop  the  proposed 
program  plans  and  documentation  at  a 
level  that  is  adequate  to  identify  risks,  de¬ 
velop  associated  management  activities 
that  they  will  use  throughout  the  program, 
and  integrate  resources,  technical  perfor¬ 
mance  measures,  and  schedule  in  the  pro¬ 
posed  program  plans.  Program  plans 
should  extend  the  CWBS  to  reflect  the 
offeror's  approach  and  include  the  sup¬ 
porting  activities,  critical  tasks,  and  pro¬ 
cesses  in  the  CWBS  dictionary.  The  associ¬ 
ated  schedules  for  each  should  be  incor¬ 
porated  into  an  integrated  master  sched¬ 
ule.  Plans  should  also  have  an  estimate  of 
the  funds  required  to  execute  the  program 
and  include  a  breakout  of  resource  require¬ 
ments  for  high  risk  areas. 
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Table  4-2.  Significant  Risks  by  Critical  Risk  Areas 


Risk  Area 

Significant  Risks 

Threat 

Uncertainty  in  threat  accuracy  and  stability. 

Sensitivity  of  design  and  technoiogy  to  threat. 

Vulnerability  of  system  to  threat  countermeasures. 

Vulnerabilityofproaram  to  inteiiigence  penetration. 

Requirements 

Operational  requirements  not  properly  established  or  vaguely  stated. 
Requirements  are  not  stable. 

Required  operating  environment  not  described. 

Requirements  do  not  address  logistics  and  suitability. 

Requirements  are  too  constrictive— identify  specific  solutions  that  force 
hiqh  cost. 

Design 

Design  implications  not  sufficiently  considered  in  concept  exploration. 
System  will  not  satisfy  user  requirements. 

Mismatch  of  user  manpower  or  skill  profiles  with  system  design  solution  or 
human-machine  interface  problems. 

Increased  skills  or  more  training  requirements  identified  late  in  the 
acquisition  process. 

Design  not  cost  effective. 

Design  relies  on  immature  technologies  or  “exotic"  materials  to  achieve 
oerformance  objectives. 

Test  and  Evaluation 

Test  planning  not  initiated  early  in  program  (Phase  0). 

Testing  does  not  address  the  ultimate  operating  environment. 

Test  procedures  do  not  address  all  major  performance  and  suitability 
specifications. 

Test  facilities  not  available  to  accomplish  specific  tests,  especially 
system-level  tests. 

Insufficient  time  to  test  thoroughly. 

Modeling  and 

Simulation 

Same  risks  as  contained  in  the  Significant  Risks  for  Test  and  Evaluation. 
M&S  are  not  verified,  validated,  or  accredited  for  the  intended  purpose. 
Program  lacks  proper  tools  and  modeling  and  simulation  capability  to 
assess  alternatives. 

Technology 

Program  depends  on  unproven  technology  for  success — ^there  are  no 
alternatives. 

Program  success  depends  on  achieving  advances  in  state-of-the-art 
technology. 

Potential  advances  in  technology  will  result  in  less  than  optimal  cost- 
effective  system  or  make  system  components  obsolete. 

Technology  has  not  been  demonstrated  in  required  operating 
environment. 

Technology  relies  on  complex  hardware,  software,  or  integration  design. 
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Table  4-2.  Significant  Risks  by  Critical  Risk  Areas 
(Continued) 


Risk  Area 

Significant  Risks 

Logistics 

Inadequate  supportability  late  in  development  or  after  fielding,  resulting  in  need 
for  engineering  changes,  increased  costs,  and/or  schedule  delays. 

Life-cycle  costs  not  accurate  because  of  poor  logistics  supportability  analyses. 
Logistics  analyses  results  not  included  in  cost-performance  tradeoffs. 

Design  trade  studies  do  not  include  supportability  considerations. 

Production/ 

Facilities 

Production  implications  not  considered  during  concept  exploration. 

Production  not  sufficiently  considered  during  design. 

Inadequate  planning  for  long  lead  items  and  vendor  support. 

Production  processes  not  proven. 

Prime  contractors  do  not  have  adequate  plans  for  managing  subcontractors. 
Sufficient  facilities  not  readily  available  for  cost-effective  production. 

Contract  offers  no  incentive  to  modernize  facilities  or  reduce  cost. 

Concurrency 

Immature  or  unproven  technologies  will  not  be  adequately  developed  before 
production. 

Production  funding  will  be  available  too  early — before  development  effort  has 
sufficiently  matured. 

Concurrency  established  without  clear  understanding  of  risks. 

Capability  of 
Developer 

Developer  has  limited  experience  in  specific  type  of  development. 

Contractor  has  poor  track  record  relative  to  costs  and  schedule. 

Contractor  experiences  loss  of  key  personnel. 

Prime  contractor  relies  excessively  on  subcontractors  for  major  development 
efforts. 

Contractor  will  require  significant  capitalization  to  meet  program  requirements. 

Cost/Funding 

Realistic  cost  objectives  not  established  early. 

Marginal  performance  capabilities  Incorporated  at  excessive  costs — ^satisfactory 
cost-performance  tradeoffs  not  done. 

Excessive  life-cycle  costs  due  to  inadequate  treatment  of  support  requirements. 
Significant  reliance  on  software. 

Funding  profile  does  not  match  acquisition  strategy. 

Funding  profile  not  stable  from  budget  cycle  to  budget  cycle. 

Schedule 

Schedule  not  considered  in  tradeoff  studies. 

Schedule  does  not  reflect  realistic  acquisition  planning. 

APB  schedule  objectives  not  realistic  and  attainable. 

Resources  not  available  to  meet  schedule. 

Management 

Acquisition  strategy  does  not  give  adequate  consideration  to  various  essential 
elements,  e.g.,  mission  need,  test  and  evaluation,  technology,  etc. 

Subordinate  strategies  and  plans  are  not  developed  in  a  timely  manner  or  based 
on  the  acquisition  strategy. 

Proper  mix  (experience,  skills,  stability)  of  people  not  assigned  to  PMO  or  to 
contractor  team. 

Effective  risk  assessments  not  performed  or  results  not  understood  and  acted 
upon. 
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The  information  required  and  the  level  of 
detail  wiU  depend  on  the  acquisition  phase, 
the  category,  and  criticality  of  the  program, 
as  well  as  on  the  contract  type  and  value. 
However,  the  detail  submitted  with  the 
proposal  must  be  at  a  sufficiently  low  level 
to  allow  identification  of  possible  conflicts 
in  the  planned  acquisition  approach  and 
to  support  the  Government's  proposal 
evaluation.  Generally,  the  CWBS  should 
be  defined  below  level  3,  by  the  contrac¬ 
tor,  only  to  the  extent  necessary  to  capture 
those  lower  level  elements  that  are  high 
cost,  high  risk,  or  of  high  management 
interest. 

4.6.5  Basis  for  Selection 

DoD  acquisition  management  must  focus  on 
balancing  performance,  schedule,  and  cost 
objectives  by  selecting  the  contractor  team 
that  provides  the  best  value  to  the  user  within 
acceptable  risk  limits.  Therefore,  the  RFP/ 
Somce  Selection  process  must  evaluate  each 
offeror's  capability  for  meeting  product  and 
process  technical,  schedule,  and  cost  require¬ 
ments  while  addressing  and  controlling  the 
risks  inherent  in  a  program. 

The  evaluation  team  should  discriminate 
among  offerors  based  upon  the  following: 

•  Risks  determined  by  comparison  with 
the  best  practices  baseline 

•  Ability  to  perform  with  a  focus  on  the 
critical  risk  elements  inherent  in  the  pro¬ 
gram 

•  Adherence  to  requirements  associated 
with  any  mandatory  legal  items 

•  Past  performance  on  efforts  similar  to 
the  proposed  program  being  evaluated. 

The  process  of  choosing  among  offerors  may 
be  enhanced  if  the  evaluation  team  includes 
risk  management  as  a  "source  selection  dis¬ 
criminator."  Risk  management  then  becomes 
an  important  factor  in  the  Source  Selection 


Authority  determination  of  who  provides  the 
most  executable  program. 

4.6.6  Source  Selection 

The  purpose  of  a  source  selection  is  to  select 
the  contractor  whose  performance  can  best 
be  expected  to  meet  the  Government's  re¬ 
quirements  at  an  affordable  price.  To  |jer- 
form  this  evaluation,  the  Government  must 
assess  both  proposal  risk  and  perjortmnce  risk 
for  each  proposal.  These  risk  assessments 
must  be  done  entirely  within  the  boundaries 
of  the  source  selection  process.  Previous  as¬ 
sessments  of  any  of  the  offerors  may  not  be 
applicable  or  allowable. 

4.6.6.1  Proposal  Risk.  This  refers  to  the 
risk  associated  with  the  offeror's  proposed 
approach  to  meet  the  Government  perfor¬ 
mance,  cost,  and  schedule  requirements. 
The  evaluation  of  proposal  risk  includes  an 
assessment  of  proposed  time  and  resources 
and  recommended  adjustments.  This  as¬ 
sessment  should  be  performed  according 
to  the  definitions  and  evaluation  standards 
developed  for  the  source  selection.  Pro¬ 
posal  risk  is,  in  essence,  a  moderate  expan¬ 
sion  of  past  evaluation  processes.  Histori¬ 
cally,  evaluators  selected  contractors  who 
demonstrated  that  they  understood  the  re¬ 
quirements  and  offered  the  best  value  ap¬ 
proach  to  meeting  the  Government's  needs. 
The  expansion  on  this  concept  is  the  spe¬ 
cific  consideration  of  risk. 

Technical  and  schedule  assessments  are 
primary  inputs  to  the  most  probable  cost 
estimate  for  each  proposal.  It  is  important 
to  estimate  the  additional  resources  needed 
to  control  any  risks  that  have  moderate  or 
high  risk  ratings.  Offerors  may  define  them 
in  terms  of  additional  time,  personnel  load¬ 
ing,  hardware,  or  special  actions  such  as 
additional  tests.  However,  whatever  the 
type  of  the  required  resources,  it  is  essen¬ 
tial  that  cost  estimates  be  integrated  and 
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consistent  with  the  technical  and  schedule 
evaluations. 

4.6.6.2  Performance  Risk.  A  performance 
risk  assessment  is  an  evaluation  of  the 
contractor's  past  and  present  performance 
record  to  establish  a  level  of  confidence  in 
the  contractor's  ability  to  perform  the  pro¬ 
posed  effort. 

A  range  of  methods  are  available  to  the  PM 
to  evaluate  performance  risk.  The  Perfor¬ 
mance  Risk  Assessment  Group  (PRAG)  is  a 
group  of  experienced  Government  person¬ 
nel  that  are  appointed  by  the  source  selec¬ 
tion  advisory  councU  Chairperson  to  permit 
performance  risk  to  be  used,  if  appropriate. 
Performance  risk  may  be  separately  assessed 
for  each  evaluation  factor  or  as  a  whole  with 
the  assessment  provided  directly  to  the 
source  selection  advisory  coundl/authority 
for  final  decision  or  indirectly  through  the 
Source  Selection  Evaluation  Board.  The  as¬ 
sessment  relies  heavily  (although  not  exclu¬ 
sively)  on  the  contractor  performance  evalu¬ 
ations  and  surveys  submitted  by  the  PMO 
and  Defense  Contract  Management  Com¬ 
mand  (DCMC). 

4.7  RISK  MANAGEMENT: 

POST-CONTRACT  AWARD 

Post-contract  award  risk  management 
builds  on  the  work  done  during  the  pre¬ 
contract  award  phase.  With  the  award  of 
the  contract,  the  relationship  between  the 
Government  and  the  contractor  changes  as 
teams  are  formed  to  address  program  risk. 
These  teams  should  validate  pre-contract 
award  management  plans  by  reviewing 
assessments,  handling  plans,  and  monitor¬ 
ing  intentions.  The  extent  of  assessments 
increases  as  the  contractor  develops  and 
refines  his  design,  test  and  evaluation,  and 
manufacturing  plans.  The  Government 
PMO  should  work  with  the  contractor  to 
refine  handling  plans. 


The  process  begins  with  an  Integrated 
Baseline  Review  dBR)  after  contract  award 
to  ensure  that  reliable  plans  and  performance 
measurement  baselines  capture  the  entire 
scope  of  work,  are  consistent  with  contract 
schedule  requirements,  and  have  adequate 
resources  assigned  to  complete  program 
tasks.  The  IBR  could  be  conducted  to  incor¬ 
porate  other  steps  identified  below.  These 
steps  suggest  an  approach  that  the  PMO 
might  take  to  initiate  the  program's  risk  man¬ 
agement  plans  and  activities  after  contract 
award.  They  are  intended  to  be  a  starting 
point,  and  the  PMO  should  tailor  the  plan  to 
reflect  each  program's  unique  needs. 

•  Conduct  initial  meeting  with  the  con¬ 
tractor  to  describe  the  program's  objectives 
and  approach  to  managing  risks.  The  PM 
may  al^  present  the  risk  management  plan. 

•  Train  members  of  PMO  and  contractor's 
organization  on  risk-management  basics,  in¬ 
corporating  the  program's  management  plan 
and  procedures  into  the  training. 

•  Review  the  pre-contract  award  risk 
plan  with  the  PMO  and  contractor,  revise 
it  as  necessary,  and  share  results  with  the 
contractor. 

•  Conduct  in-depth  review  of  the  pre¬ 
contract  award  risk  assessments  and  ex¬ 
pand  the  review  to  include  any  new  infor¬ 
mation  obtained  since  the  award  of  the  con¬ 
tract. 

•  Review  and  revise  risk-handling  plans 
to  reflect  the  reassessment  of  risks. 

•  Review  the  program's  documentation 
requirements  with  the  contractor.  Ensure 
that  the  PMO  and  contractor  understand 
the  purpose,  format,  and  contents  of  vari¬ 
ous  risk  reports. 

•  Initially,  it  may  be  necessary  to  estab¬ 
lish  a  formalized  PMO-contractor  risk 
management  organization  for  the  program, 
consistent  with  the  terms  of  the  contract. 
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•  Working  with  the  contractor,  refine  the 
risk-monitoring  plans  and  procedures. 

•  Establish  the  program  reporting  require¬ 
ments  with  the  contractor.  Describe  the  risk 
management  information  system  that  the 
program  has  established,  including  proce¬ 
dures  for  providing  information  for  data 
entry,  and  identify  reports  for  the  PMO  and 
contractor. 

•  In  conjunction  with  the  contractor, 
identify  other  risk-management  activities 
that  need  to  be  performed. 

•  Manage  the  program  risk  in  accor¬ 
dance  with  the  risk  management  plan  and 
contract. 

•  Working  with  the  contractor,  refine  the 
lisk-monitoring  plans  and  procedures  and 
develop  appropriate  measures  and  metrics 
to  track  moderate  and  high  risk  items. 

4.8  RISK  MANAGEMENT 
REPORTING  AND 
INFORMATION  SYSTEM 

The  PMO  should  have  a  practical  method  for 
risk-management  reporting,  and  an  informa¬ 
tion  system  that  supports  a  risk-management 
program.  The  reporting  needs  of  the  PM  es¬ 
tablish  the  type,  format,  and  frequency  of  in¬ 
formation  sharing.  The  IPT  concept  suggests 
that  the  entire  acquisition  program  team 
needs  access  to  the  risk  management  infor¬ 
mation,  and  the  prime  contractor(s)  should 
have  access  to  information,  consistent  with 
acquisition  regulations.  The  reporting  and 
information  system  chosen  may  be  Govern¬ 
ment-  or  contractor-owned.  See  Chapter  5 
for  an  example  of  an  MIS. 

4.9  RISK  MANAGEMENT  TRAINING 

A  successful  management  program  de¬ 
pends,  to  a  large  extent,  on  the  level  of  risk 
management  training  the  PMO  members 
and  the  functional  area  experts  receive.  The 
training  will  prepare  them  for  critical  tasks. 


such  as  risk  assessments.  DoD  schools  of¬ 
fer  some  risk-management  training;  how¬ 
ever,  PMs  will  need  to  organize  and  con¬ 
duct  principal  training  for  the  program  of¬ 
fice.  A  three-part  framework  for  training 
covers  program-specific  risk  management 
issues,  general  structure  and  process,  and 
techniques. 

(1)  The  program-specific  training 
should  ensure  that  everyone  has  a  common 
vision.  It  should  cover  the  acquisition  strat¬ 
egy,  the  companion  risk-management  plan, 
the  PM's  risk-management  structure  and 
associated  responsibilities,  and  the  MIS. 

(2)  The  following  topics  provide  a  start¬ 
ing  point  for  general  training  syllabus  de¬ 
velopment.  The  final  syllabus  should  be 
tailored  to  meet  the  program's  specific 
needs.  Table  4-3  provides  a  list  of  refer¬ 
ences  that  will  be  useful  in  developing  the 
syllabus  and  lesson  plans. 

•  Concept  of  Risk 

•  Risk  Planning 

•  Risk  Identification 

•  Risk  Analysis  (as  applicable) 

•  Risk  Handling 

•  Risk  Monitoring. 

(3)  The  third  area  of  training  concerns 
risk  management  techniques,  concentrating 
on  the  techniques  the  PMO  plans  to  employ. 
The  training  should  focus  on  how  to  use  the 
techniques  and  should  include  examples  of 
their  use.  Chapter  5,  Risk  Management  Tech- 
nicjues,  of  this  Guide  provides  a  starting  point. 
It  contains  a  general  discussion  of  a  set  of 
techniques  that  address  all  elements  of  the 
risk  management  process.  The  discussion  of 
each  technique  contains  a  list  of  references 
that  provide  a  more  in-depth  description  of 
the  technique.  The  set  of  techniques  is  not 
exhaustive  and  the  program  office  should 
add  to  the  list,  if  necessary. 
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Table  4-3.  Risk  Management  Reference  Documents 


Document 

Description 

DoD  4245.7-M,  Transition  from  Development  to 
Production,  September  1985. 

Provides  a  structure  for  identifying  technical  risk 
areas  in  the  transition  from  a  program's 
development  to  production  phases.  The  structure 
is  geared  toward  development  programs  but,  with 
modifications,  could  be  used  for  any  acquisition 
program.  The  structure  identifies  a  series  of 
templates  for  each  of  the  development 
contractor's  critical  engineering  processes.  The 
template  includes  potential  areas  of  risk  and 
methods  for  reducing  risk  in  each  area. 

Risk  Management  Concepts  and  Guidance, 
Defense  Systems  Management  College,  March 
1989. 

Devoted  to  various  aspects  of  risk  management. 

Systems  Engineering  Management  Guide, 

Defense  Systems  Management  College,  January 
1990,  Section  15. 

Devoted  to  risk  analysis  and  management  and 
provides  a  good  overview  of  the  risk  management 
process. 

Continuous  Risk  Management  Guidebook, 

Software  Engineering  Institute,  Carnegie  Mellon 
University,  1996. 

Provides  a  risk  management  methodology  similar 
to  the  one  described  in  the  Deskbook.  Its  value  is 
that  it  subdivides  each  process  into  a  series  of 
steps;  this  provides  useful  insights.  Appendix  A 
describes  40  risk-management  techniques,  the 
majority  of  which  are  standard  management 
techniques  adapted  to  risk  management.  This 
makes  them  a  useful  supplement  to  the  Deskbook 
identified  techniques. 

A  Systems  Engineering  Capability  Maturity  Model, 
Version  1 .0  Software  Engineering  Institute 
(Carnegie  Mellon  University),  Handbook  SECMM- 
94-04,  December  1994. 

Describes  one  approach  to  conducting  an  Industry 
Capabilities  Review.  Section  PA  10  (pp.  4-72  - 
4-76)  discusses  software  risk  management.  The 
material  presented  in  this  handbook  also  can  be 
tailored  to  apply  to  system  and  hardware  risk 
management. 

Software  Acquisition  Capability  Maturity  Model, 
Version  1 .01  Software  Engineering  Institute 
(Carnegie  Mellon  University),  Technical  Report, 
December  1996. 

Describes  an  approach  to  assess  the  software 
acquisition  processes  of  the  acquiring 
organization  and  identifies  areas  for 
improvement. 

Capability  Maturity  Model  for  Software  (SM- 
CMM),  Version  1.1,/CMU/SEI-93-TR-24  February 
1993. 

This  is  a  tool  that  allows  an  acquiring  organization 
to  assess  the  software  capability  maturity  of  an 
organization. 

Taxonomy-Based  Risk  Identification,  Software 
Engineering  Institute,  Carnegie  Mellon  University, 
CMU/SEI-93-TR-6  (ESC-TR-93-183),  June  1993. 

Describes  a  method  for  facilitating  the  systematic 
and  repeatable  identification  of  risks  associated 
with  the  development  of  a  software-intensive 
project.  This  method  has  been  tested  in  active 
Government-funded  defense  and  civilian  software 
development  projects.  The  report  includes 
macro-level  lessons-learned  from  the  field  tests. 

NAVSO  P-6071: 

Navy  "best  practices"  document  with 
recommended  implementations  and  further 
discussion  on  the  material  in  DoD  4245.7-M. 
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Table  4-3.  Risk  Management  Reference  Documents 
(Continued) 


Document 

Description 

Risk  Management,  AFMC  Pamphlet  63-101  July 
1997. 

An  excellent  pamphlet  on  risk  management  that 
is  intended  to  provide  PMs  and  the  PMO  with  a 
basic  understanding  of  the  terms,  def  initions,  and 
processes  associated  with  effective  risk 
management.  It  is  very  strong  on  how  to  perform 
pre-contract  award  risk  management. 

Acquisition  Software  Development  Capability 
Evaluation,  AFMC  Pamphlet  63-103, 15  June  94. 

Describes  one  approach  to  conducting  an  Industry 
Capabilities  Review.  This  two-volume  pamphlet 
was  generated  from  material  originated  at 
Aeronautical  Systems  Center.  The  concepts 
support  evaluations  during  source  selection  and 
when  requested  by  IPTs.  The  material  presented 
in  this  pamphlet  also  can  be  tailored  to  apply  to 
system  and  hardware  risk  management. 
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RISK  MANAGEMENT  TECHNIQUES 


5.1  INTRODUCTION 

This  Chapter  provides  top-level  informa¬ 
tion  on  a  number  of  techniques  currently 
used  in  DoD,  and  a  combination  of  tech¬ 
niques  used  by  the  Services,  industry,  and 
academia.  Collectively,  they  focus  on  the 
components  of  the  risk  management  pro¬ 
cess  and  address  critical  risk  areas  and  pro¬ 
cesses.  The  write-ups  describe  the  tech¬ 
niques  and  give  information  on  their  ap¬ 
plication  and  utility.  The  descriptions  are 
at  a  level  of  detail  that  should  permit  po¬ 
tential  users  to  evaluate  the  suitability  of 
the  techniques  for  addressing  their  needs; 
however,  the  material  does  not,  in  most 
cases,  provide  all  the  information  that  is 
required  to  use  a  technique.  Readers  will 
find  that  if  a  particular  technique  looks 
promising,  they  can  obtain  enough  infor¬ 
mation  from  the  references  and  tools  that 
will  enable  program  offices  to  apply  them. 
The  descriptions  are  in  a  format  that  aids 
comparison  with  other  approaches. 

5.2  OVERVIEW 

Techniques  are  available  to  support  risk 
management  activities.  None  are  required 
by  DoD,  but  some  have  been  successfully 
used  in  the  past  by  DoD  PMs.  Many  of  the 
techniques  support  processes  that  are  part 
of  sound  management  and  systems  engi¬ 
neering  and  give  Government  and  contrac¬ 
tor  PMs  the  tools  for  considering  risk  when 
making  decisions  on  managing  the  pro¬ 
gram. 


Several  tools  are  available  to  support  each 
of  the  components  of  the  risk  management 
process,  i.e.,  planning,  assessing,  handling, 
and  monitoring  and  documenting.  Al¬ 
though  tool  developers  may  claim  other¬ 
wise,  none  are  integrated  to  totally  satisfy 
all  needs  of  a  PM.  Most  likely,  a  PM  will 
choose  an  overall  risk  strategy,  write  a  plan 
to  reflect  his  strategy,  review  the  list  of 
proven  techniques  to  support  the  compo¬ 
nents  of  risk  management,  assess  the  tech¬ 
niques  against  the  program's  needs  and 
available  resources,  tailor  the  techniques  to 
suit  the  needs  of  the  program,  and  train 
program  office  members  to  implement  the 
plan. 

5.3  RISK  PLANNING  TECHNIQUES 
5.3.1  Description 

This  technique  suggests  an  approach  to  risk 
planning;  the  process  of  developing  and 
documenting  an  organized,  comprehensive 
approach.  It  also  suggests  interactive  strat¬ 
egy  and  methods  for  identifying  and  track¬ 
ing  risk  drivers,  developing  risk-handling 
plans,  performing  continuous  assessments 
to  determine  how  risks  have  changed,  and 
planning  adequate  resources.  The  risk 
planning  technique  is  applicable  to  all  func¬ 
tional  areas  that  compose  the  program,  es¬ 
pecially  critical  areas  and  processes.  Us¬ 
ing  the  acquisition  strategy  as  a  starting 
point  results  in  the  development  of  a  pro¬ 
gram  risk  management  strategy,  from 
which  flows  a  management  plan  that  pro¬ 
vides  the  detailed  information  and  direc- 


an  effective  management  program.  This  risk 
management  plan  provides  the  PM  with  an 
effective  method  to  define  a  program,  one 
that  fixes  responsibility  for  the  implementa¬ 
tion  of  its  various  aspects,  and  supports  the 
acquisition  strategy. 

The  technique  should  first  be  used  in  Phase 
0  following  the  development  of  the  initial 
acquisition  strategy.  Subsequently,  it  may 
be  used  to  update  the  management  plan 
on  the  following  occasions:  (1)  whenever 
the  acquisition  strategy  changes,  or  there 
is  a  major  change  in  program  emphasis;  (2) 
in  preparation  for  major  decision  points; 
(3)  in  preparation  for  and  immediately  fol¬ 
lowing  technical  audits  and  reviews;  (4) 
concurrent  with  the  review  and  update  of 
other  program  plans;  and  (5)  in  prepara¬ 
tion  for  a  PMO  submission. 


The  PMO  risk  management  coordinator,  if 
assigned,  develops  the  risk  management  plan 
based  on  guidance  provided  by  the  PM.  To 
be  effective,  the  PM  must  make  risk  manage¬ 
ment  an  important  program  management 
function  and  must  be  actively  involved  in  the 
risk  planning  effort.  Planning  requires  the 
active  participation  of  essentially  the  entire 
PMO  and  contractor  team. 

5.3.2  Procedures 

Figure  5-1  graphically  depicts  the  process 
to  be  followed  in  applying  this  technique. 
The  procedure  consists  of  a  number  of  it¬ 
erative  activities  that  result  in  the  devel¬ 
opment  of  the  risk  management  strategy 
and  a  risk  management  plan. 

The  acquisition  strategy  and  related  man¬ 
agement  planning  efforts  (program  man- 


PM  Guidance 

1  ^ 

INPUT 

•  Evaluate  risk  planning 
requirements 

•  Assess  the  program’s  current  risk 
situation 

•  Develop  a  risk  management 

strategy 

•  Acquisition  strategy 

•  Prior  risk  management 

plan  (if  any)  - ^ 

•  Known  risks 

•  System  description 

•  Program  description 

•  Determine  the  tasks  and  guidance 
required  to  implement  the  risk 
management  strategy 

•  Develop  the  PMO’s  approach  to 
risk  management  in  general 

•  Provide  application  guidance  for 

OUTPUT 

^  •  Risk 
^  Management 
Plan 

risk  management  component 
processes 

•  Develop  inputs  for  other  acquisition 

strategies 

t 

•  Program-Level  IPT 

•  Risk  management  coordinator 

Figure  5-1.  Risk  Planning  Technique  Inputs  and  Output 
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agement,  and  systems  engineering),  pro¬ 
gram  constraints,  and  any  existing  risk 
management  planning  are  integrated  and 
evaluated  in  the  context  of  the  PM's  guid¬ 
ance,  which  provides  the  direction  for  the 
planning  process.  Typical  types  of  PM 
guidance  are  concerns  about  certain  catego¬ 
ries  of  risk,  guidance  on  funding  of  han¬ 
dling  activities,  emphasis  to  be  placed  on 
risk  management  training,  and  frequency 
and  type  of  internal  reports. 

The  integration  and  evaluation  of  the  pri¬ 
mary  inputs  establish  the  requirements  and 
scope  of  the  planning  effort  through  an  as¬ 
sessment  of  the  program's  current  risk  situa¬ 
tion.  The  results  of  the  assessment  provide 
the  basis  for  development  of  management 
strategy.  The  strategy  should  reflect  the  level 
of  risk  that  the  PM  is  prepared  to  accept,  and 
should  provide  guidance  on  how  and  when 
known  risks  will  be  brought  under  control. 
It  should  also  describe  the  risk  management 
process  the  PMO  will  employ  and  the  orga¬ 
nization  and  structure  of  the  management 
program,  addressing  things  such  as  risk  rat¬ 
ings,  the  use  of  an  MIS,  policy  and  proce¬ 
dures  on  sharing  of  risk  management  infor¬ 
mation,  and  training. 

The  PMO  should  consider  creating  an  MIS 
early  in  the  planning  process.  It  will  serve  as 
a  planning  source  and  the  data  may  be  used 
for  creating  reports.  It  will  also  become  the 
repository  for  all  current  and  historical  in¬ 
formation  related  to  risk.  Eventually,  this 
information  may  include  risk  assessment 
documents,  contract  deliverables,  if  appro¬ 
priate,  and  other  risk-related  reports. 

Based  on  the  management  strategy,  the  plan 
identifies  specific  tasks  to  be  accomplished 
and  assigns  responsibility  for  their  execution. 
The  timing  of  these  tasks  should  be  incorpo¬ 
rated  into  an  integrated  master  schedule. 
Guidance  for  task  execution  and  control 
should  also  be  developed,  covering  such 


things  as  the  suggested  techniques  to  be  used 
for  each  component,  any  assistance  available 
to  Sub-Tier  IPTs,  the  use  of  funds,  the  policy 
on  the  use  of  independent  risk  assessors,  etc. 
This  information  may  be  documented  in  a 
form  such  as  a  risk  management  plan.  A 
sample  format  for  the  Risk  Management  Plan 
is  shown  in  Figure  5-2.  Appendix  B  is  an 
example  of  a  Risk  Management  Plan. 

The  contents  of  the  risk  management  strat¬ 
egy  and  plan  should  be  consistent  with  the 
acquisition  strategy  and  other  program  plans 
derived  from  the  acquisition  strategy.  This 
will  help  to  ensure  that  risk  is  considered  in 
all  program  activities  and  that  it  does  not 
become  a  "stove  pipe"  function. 

5.4  RISK  ASSESSMENT 
TECHNIQUES 

5.4.1  Product  (WBS)  Risk  Assessment 

5.4.1.1  Description.  This  technique  iden¬ 
tifies  those  risks  associated  with  a  given 
system  concept  and  design.  The  difference 
between  the  process  (DoD  4245.7-M)  tech¬ 
nique  and  this  approach  is  that  DoD  4245.7- 
M  addresses  the  contractor's  engineering 
and  manufacturing  process  and  this  tech¬ 
nique  focuses  on  the  resulting  product.  This 
technique  is  used  to  identify  and  analyze 
risks  in  the  following  critical  risk  areas: 
design  and  engineering,  technology,  logis¬ 
tics,  concurrency  and  manufacturing. 

The  WBS  serves  as  the  starting  point  to  de¬ 
scribe  contract  work  that  will  be  done  and 
the  resulting  product  and  determines  the 
risk  events  in  each  critical  risk  area.  The 
risk  events — events  that  might  have  a  det¬ 
rimental  impact  on  the  system,  subsystems, 
or  components — ^are  evaluated  to  identify 
and  characterize  specific  risks. 

This  technique  should  be  used  shortly  af¬ 
ter  the  completion  of  the  prime  contractor's 
WBS.  Thereafter,  it  should  be  used  regu- 
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INTRODUCTION.  This  section  should  address  the  purpose  and  objective  of  the  plan,  and  provide  a  brief 
summary  of  the  program,  to  include  the  approach  being  used  to  manage  the  program,  and  the  acquisition 
strategy. 

PROGRAM  SUMMARY.  This  section  contains  a  brief  description  of  the  program,  inctuding  the  acquisitim 
strategy  and  the  program  management  approach.  The  acquisition  strategy  shouid  address  its  linkage  to  the  risk 
management  strategy. 

DEFINITIONS.  Definitions  used  by  the  program  office  should  be  consistent  with  DoD  definitions  for  ease  of 
understanding  and  consistency.  However,  the  DoD  definitions  allow  program  managers  flexibility  in  constructing 
their  risk  management  programs.  Therefore,  each  program's  risk  management  plan  may  include  definitions  that 
expand  the  DoD  definitions  to  fit  its  particular  needs.  For  example,  each  plan  should  include,  among  other  things, 
definitions  for  the  ratings  used  for  technical,  schedule  and  cost  risk.  (Discussion  of  risk  rating  is  contained  in 
Acquisition  Deskbook  Section  2.5.2. 1.) 

RISK  MANAGEMENT STRA TEGY  AND  APPROACH.  Provide  an  overview  of  the  risk  management  approach, 
to  include  the  status  of  the  risk  management  effort  to  date,  and  a  description  of  the  program  risk  management 
strategy.  See  Acquisition  Deskbook  Sectbns  2.5.2. 1  and2.5.2.3. 

ORGANIZATION.  Describe  the  risk  management  organizatbn  of  the  program  office  and  Tist  the  responsbilities 
of  each  of  the  risk  management  participants.  See  Acquisition  Deskbook  Section  2.5.2.3. 

RISK  MANAGEMENT  PROCESS  AND  PROCEDUI^S.  Describe  the  program  risk  management  process  to  be 
en^)loyed,  i.e.,  risk  planning,  assessment,  handling,  monitoring  and  documentation,  and  a  basic  explanation  of 
these  components.  See  Acquisition  Deskbook  Section  2.5.2. 1.  Also  provbe  application  gubance  for  each  of  the 
risk  management  functions  In  the  process.  If  possible,  the  guidance  should  be  as  general  as  possible  to  allow  the 
program’s  risk  management  organization  (e.g.,  IPTs)  flexibility  in  managing  the  program  risk,  yet  specific  enough 
to  ensure  a  common  and  coordinated  approach  to  risk  management  It  should  address  how  the  Information 
associated  with  each  element  of  the  risk  management  process  will  be  documented  and  made  available  to  all 
participants  in  the  process,  and  how  risks  will  be  tracked,  to  Include  the  identification  of  specific  metrics  if 
possible. 

RISK  PLANNING.  This  section  describes  the  risk  planning  process  and  provides  guidance  on  how  it  will  be 
accomplished,  and  the  relationship  between  continuous  risk  planning  and  this  RMP.  Guidance  on  updates  of  the 
RMP  and  the  approval  process  to  be  foibwed  should  also  be  included.  See  Section  2.5.2. 1  of  the  Deskbook  for 
information  on  risk  planning. 

RISK  ASSESSMENT.  This  section  of  the  plan  describes  the  assessment  process  and  procedures  for  examming 
the  critical  risk  areas  and  processes  to  identify  and  document  the  assocbted  risks.  It  also  summarizes  the 
analyses  process  for  each  of  the  risk  areas  leading  to  the  determination  of  a  risk  rating.  This  rating  is  a  reflection 
of  the  potential  impact  of  the  risk  In  terms  of  its  variance  from  known  Best  Practices  or  probability  of  occurrence, 
its  consequence,  and  its  relationship  to  other  risk  areas  or  processes.  This  section  may  include: 

•  Overview  and  scope  of  the  asses&nent  process 

•  Sources  of  information 

•  Information  to  be  reported  and  forma  ts 

•  Description  of  how  risk  information  Is  retained 

•  Assessment  techniques  and  tools  (see  Section  2.5.2.4.2  of  the  Deskbook) 

RISK  HANDLING.  This  section  describes  the  procedures  that  can  be  used  to  determine  and  evaluate  various 
risk  handling  options,  and  identifies  tools  that  can  assist  in  implementing  the  risk  handling  process.  It  also 
provides  guidance  on  the  use  of  the  various  handling  options  for  specific  risks. 

RISK  MONITORING.  This  section  describes  the  process  and  procedures  that  will  be  foibwed  to  monitor  the 
status  of  the  various  risk  events  identified.  It  should  provide  criteria  for  the  selection  of  risks  to  be  reported  on, 
and  the  frequency  of  reporting.  Guidance  on  the  selection  of  metrics  should  also  be  included. 

RISK  MANAGEMENT  INFORMATION  SYSTEM,  DOCUMENTATION  AND  REPORTS.  This  section  describes 
the  MIS  structure,  rules,  and  procedures  that  will  be  used  to  document  the  results  of  the  risk  management 
process.  It  also  identifies  the  risk  management  documentation  and  reports  that  will  be  prepared;  specifies  the 
format  and  frequency  of  the  reports;  and  assigns  responsibility  for  their  preparation. 


Figure  5-2.  Sample  Format  for  Risk  Management  Plan 
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ter  the  completion  of  the  prime  contractor's 
WBS.  Thereafter,  it  should  be  used  regu¬ 
larly  up  to  the  start  of  production.  The  tech¬ 
nique  can  be  used  independently  or  in  con¬ 
junction  with  other  risk  assessment  tech¬ 
niques,  such  as  the  Process  (DoD  4245.7- 
M)  Risk  Assessment  technique.  It  may,  if 
appropriate,  also  be  used  in  conjunction 
with  the  Integrated  Baseline  Review  (IBR), 
which  is  conducted  immediately  after  con¬ 
tract  award.  See  Section  1.4.2.4.3  of  the 
Deskbook  for  a  discussion  of  IBR.  A  World 
Wide  Web  Site  is  also  available  at 
www.acq.osd.mil./pm/ ibrmats.htm. 

To  apply  this  technique,  joint  Government 
and  industry  evaluation  teams  should  exam¬ 
ine  the  appropriate  WBS  levels  in  each  Sub- 
Tier  BPTs  product  area.  If  necessary,  comple¬ 
mentary  indusby-only  teams  may  take  an  in- 
depth  look  at  selected  areas  at  lower  WBS 
levels.  At  times,  it  may  be  desirable  to  in¬ 
clude  outside  industry  experts  on  the  teams 
to  aid  in  the  examination  of  specific  WBS  el¬ 
ements  or  functional  areas. 

5.4.1.2  Procedures.  Figure  5-3  depicts  the 
process  used  in  this  technique.  The  first 


step  is  to  review  the  WBS  elements  down 
to  the  level  being  considered,  and  segment 
them  into  risk  events.  This  review  should 
consider  the  critical  areas  (design  and  en¬ 
gineering,  technology,  logistics,  etc.),  and 
their  elements  that  may  help  to  describe 
risk  events.  Table  5-1  shows  a  partial  list¬ 
ing  of  these  elements. 

Using  information  from  a  variety  of 
sources,  such  as  program  plans,  prior  risk 
assessments,  expert  interviews,  etc.,  the 
risk  events  are  examined  to  identify  spe¬ 
cific  risks  in  each  critical  area.  They  are 
then  analyzed  to  determine  probability  of 
occurrence  and  consequences,  along  with 
any  interdependencies.  Several  techniques 
and  tools  are  available  to  accomplish  this, 
including,  among  others,  technology  as¬ 
sessments,  modeling  and  simulation,  haz¬ 
ard  analysis,  and  fault  tree  analysis. 

The  results  of  this  analysis  should  be  docu¬ 
mented  in  a  program-specific  standard  for¬ 
mat,  such  as  a  Risk  Information  Form  (RIF). 
The  risks,  along  with  others  identified  us¬ 
ing  other  techniques,  can  be  prioritized  and 
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Figure  5-3.  Product  (WBS)  Risk  Assessment  Technique  Inputs  and  Outputs 
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Table  5-1.  Critical  Risk  Areas  and  Example  Elements 


Design  and 
Engineering 

Design/technology  approach 

Operational  environments 
External/internal  interfaces 

Use  of  standard  parts/program  parts  list 

System/subsystem  critical  design 
requirement 

Integration  requirements 

Human-machine  interface 

Design  growth  capacity 

Design  maturity 

Safety  &  health  hazards 

Manpower,  training  and  skill  profiles 

Training 

Technologies  supportability 

Manufacturing  technologies 

Logistics 

Operations  and  Maintenance  (O&M) 
concept 

System  diagnostic  requirement 

Repairability  and  Maintainability  (R&M) 
requirements 

Supply  support  requirements 

Built-in  test  (BIT)  requirements 

Support  equipment  requirements 
Maintenance  interfaces 

Level  of  repair  decisions 

Training  equipment  design 

Testing 

Integrated  test 

Qualification  testing 

Subsystem  test  limits 

Test  environmental  acceleration 
Supportability  test  results 

Manufacturing 

Design  producibility 

Manufacturing  capability  requirements 
Parts/assemblies  availability 

Special  tooling/test  equipment  planning 
Process/tooling  proofing 

Production  equipment  availability 

Concurrency 

Program  schedule  adequacy 

Development  phases  concurrency 

aggregated  using  the  technique  described 
later  in  this  chapter. 

5.4.2  Process  (DoD  4245.7-M) 

Risk  Assessment 

5.4.2.1  Description.  This  technique  is  used 
to  assess  (identify  and  analyze)  program 
technical  risks  resulting  from  the 
contractor's  processes.  It  is  based  on  the 
application  of  the  technical  risk  area  tem¬ 
plates  found  in  DoD  4245.7-M.  These  tem¬ 
plates  describe  the  risk  areas  contained  in 
the  various  technical  processes  (e.g.,  de¬ 
sign,  test,  production,  etc.)  and  specify 
methods  for  reducing  risks  in  each  area. 
Success  of  any  risk  reduction  efforts  asso¬ 
ciated  with  this  technique  will  depend  on 
the  contractor's  ability  and  willingness  to 


make  a  concerted  effort  to  replace  any  de¬ 
ficient  engineering  practices  and  proce¬ 
dures  with  best  industrial  practices. 

One  of  the  primary  benefits  of  this  tech¬ 
nique  is  that  it  addresses  pervasive  and 
important  sources  of  risk  in  most  DoD  ac¬ 
quisition  programs  and  uses  fundamental 
engineering  principles  and  proven  proce¬ 
dures  to  reduce  technical  risks.  The  tech¬ 
nique  is  accepted  by  many  aerospace  com¬ 
panies  in  normal  business  activities,  and 
in  fact,  was  developed  by  a  group  of  Gov¬ 
ernment  and  aerospace  experts. 

The  technique  is  applicable  during  Phases  0, 
I,  and  n  of  program  development.  The  de¬ 
scription  of  each  template  in  DoD  4245.7-M 
shows  the  phases  in  which  the  template 
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should  be  applied.  The  specific  timing  of  the 
application  within  the  phases  should  be  de¬ 
termined  based  on  the  type  of  program,  the 
acquisition  strategy  and  plans,  and  the  judg¬ 
ment  of  program  officials.  It  should  also  be 
used  in  preparation  for  milestone  decisions 
and  when  preparing  for  source  selection. 
This  technique  may  be  used  independently 
or  in  conjunction  with  other  risk  assessment 
techniques.  When  feasible,  a  Govemment- 
industiy  evaluation  team  should  be  formed 
early  in  the  program  to  apply  this  technique. 

5A.2.2  Procedures.  Figure  5-4  shows  the 
basic  approach  used  in  this  technique.  The 
DoD  4245.7-M  templates  are  used  in  con¬ 
junction  with  the  contract  requirements 
and  specifications  to  identify  those  techni¬ 
cal  processes  critical  to  the  program  and  to 
establish  a  program  baseline  of  contractor 
processes.  When  possible,  the  program 


baseline  should  be  determined  by  evalu¬ 
ating  actual  contractor  performance,  as 
opposed  to  stated  policy.  For  example, 
design  policy  should  be  determined  from 
interviewing  designers  and  not  simply 
from  reviewing  written  corporate  policies. 

This  program  baseline  should  then  be  com¬ 
pared  to  a  baseline  of  industry-wide  pro¬ 
cesses  and  practices  that  are  critical  to  the 
program.  The  baseline  should  be  developed 
by  reviewing  and  compiling  known  best 
practices  in  use  by  various  companies  in  both 
defense  and  non-defense  sectors.  One  source 
of  best  practices  information  is  the  Program 
Manager's  Work  Station  (PMWS),  a  series  of 
PC  expert  systems  designed  to  aid  in  the 
implementation  of  DoD 4245.7-M.  The  point 
of  contact  for  the  PMWS  is  the  Best  Manage¬ 
ment  Practices  Center  of  Excellence  [tele¬ 
phone  (301)  403-8100]. 
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The  differences  between  the  two  baselines 
are  a  reflection  of  the  technical  process  risk 
present.  These  results  should  be  docu¬ 
mented  in  a  standard  format,  such  as  a  pro- 
gram-specific  Risk  Information  Form  (see 
MIS  discussion  this  section)  to  facilitate  the 
development  of  a  risk  handling  and  risk 
reporting  plan. 

5.4.3  Program  Documentation 

Evaluation  Risk  Identification 

5.43.1  Description.  This  technique  provides 
a  methodology  for  comparing  key  program 
documents  and  plans  to  ensure  that  they  are 
consistent  and  traceable  to  one  another.  Pro¬ 
gram  documents  and  plans  are  hierarchical 
in  nature.  If  the  contents  (activities,  events, 
schedules,  requirements,  specifications,  etc.) 
of  a  document  or  plan  do  not  flow  from  or 
support  the  contents  of  those  above,  below, 
or  adjacent  to  it,  there  is  a  strong  chance  that 
risk  win  be  introduced  into  the  program  or 
that  known  risks  wiU  not  be  adequately  ad¬ 
dressed.  This  technique  reduces  those  risks 
and  improves  the  quahty  of  program  docu¬ 
mentation. 

This  technique  can  be  used  in  any  acquisi¬ 


tion  phase  as  documents  or  plans  are  being 
developed  or  updated.  The  comparison  of 
program  documentation  and  plans  should 
be  performed  by  a  smaU  team  of  experienced, 
knowledgeable  personnel  who  are  intimately 
familiar  with  the  total  program. 

5.4.3.2  Procedures.  Figure  5-5  shows  the 
process  used  in  this  technique.  The  primary 
inputs  to  the  process  are  the  PMO  documents 
that  detail  the  steps  involved  in  executing  the 
program.  These  include,  for  example,  the 
Mission  Need  Statement  (MNS),  ORD,  ac¬ 
quisition  plan,  any  master  management  plan. 
Test  and  Evaluation  Master  Plan  (TEMP), 
manufacturing  plan,  etc.  Another  set  of  key 
input  documents  are  those  used  to  commu¬ 
nicate  with  the  prime  contractor,  e.g.,  WBS, 
specifications.  Statement  of  Work  (SOW)  or 
equivalent  such  as.  Statement  of  Objectives, 
etc.  Before  any  comparison,  the  PMO  should 
review  all  documents  for  accuracy  and  com¬ 
pleteness  and  establish  correlations  among 
them.  Figure  5-6  shows  an  example  of  the 
type  of  correlation  that  should  exist  among 
the  MNS,  ORD,  and  TEMP  during  Phase  0. 

If  the  comparison  shows  any  gaps  or  in- 
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Figure  5-5.  Plan  Evaluation  Technique  Inputs  and  Output 
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Figure  5-6.  Phase  0  Correlation  of  Selected  Documents  (Example) 


consistencies,  reviewers  should  identify 
them  as  possible  risks  on  a  risk  identifica¬ 
tion  form,  the  output  of  this  process. 

5.4.4  Threat  and  Requirements  Risk 
Assessment 

5.4.4.1  Description.  This  technique  de¬ 
scribes  an  approach  to  assess  risks  associ¬ 
ated  with  requirements  and  threat  and  to 
identify  requirements  and  threat  elements 
that  are  risk  drivers.  Because  operational 
needs,  environmental  demands,  and  threat 
determine  system  performance  require¬ 
ments,  to  a  large  degree,  they  are  a  major 
factor  in  driving  the  design  of  the  system 
and  can  introduce  risk  in  a  program.  Fur¬ 
ther,  with  the  introduction  of  CAIV,  PMs 
and  users  are  directed  to  examine  perfor¬ 
mance  requirements  and  identify  areas  that 
are  not  critical  and  are  available  for  trade 
to  meet  cost  objectives.  Risk  is  a  factor  in 
CAIV  considerations. 

The  requirements  assessment  process  fo¬ 
cuses  on:  determining  if  operational  re¬ 
quirements  are  properly  established  and 
clearly  stated  for  each  program  phase;  en¬ 
suring  that  requirements  are  stable  and  the 


operating  environment  is  adequately  de¬ 
scribed;  addressing  logistics  and  suitabil¬ 
ity  needs;  and  determining  if  requirements 
are  too  constrictive,  thereby  identifying  a 
specific  solution.  The  threat  assessment 
process  addresses  uncertainty  in  threat  ac¬ 
curacy  and  stability,  sensitivity  of  design 
and  technology  to  threat,  vulnerability  of 
the  system  to  threat  countermeasures,  and 
vulnerability  of  the  program  to  intelligence 
penetration.  PMs  should  view  require¬ 
ments  in  the  context  of  the  threat  and  ac¬ 
curately  reflect  operational,  environmen¬ 
tal,  and  suitability  requirements  in  design 
documents. 

PMs  should  use  threat  and  requirements 
assessments  during  the  early  phases  of  pro¬ 
gram  development  and,  as  necessary,  as  the 
program  advances  through  development. 
Early  and  complete  understanding  of  the 
requirements  and  threat  precludes  misun¬ 
derstandings  between  the  requirements 
and  development  communities,  helps  to 
identify  risk  areas,  and  allows  early  plan¬ 
ning  to  handle  risk.  Consequently,  the  user 
should  be  actively  involved  in  this  process 
from  the  beginning. 
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Figure  5-7.  Threat  and  Requirement  Risk  Assessment  Technique  Inputs  and  Outputs 


5AA.2  Procedures.  Figure  5-7  depicts  the 
process  used  in  this  technique.  The  basic 
approach  is  to  conduct  a  thorough  review 
of  the  documents  containing  performance 
requirements  and  threat  information,  e.g., 
ORD,  TEMP,  System  Specification,  Special 
Threat  Assessment  Report  (STAR),  Design 
Reference  Mission  Profile,  etc.,  to  deter¬ 
mine  stability,  accuracy,  operating  environ¬ 
ment,  logistics  and  suitability  require¬ 
ments,  and  consistency  between  these  re¬ 
quirements  and  the  threat  considerations 
cited  above.  There  should  be  an  under¬ 
standing  between  the  users  and  the  devel¬ 
opers  on  Key  Performance  Parameters 
(KPPs)  in  order  to  identify  the  require¬ 
ments  that  are  most  important  and  critical 
to  program  success.  The  Design  Reference 
Mission  Profile  and  Design  Requirements 
templates  in  DoD  4245.7-M  and  the  Pro¬ 
gram  Documentation  Evaluation  Risk 
Identification  technique  may  be  useful  in 
support  of  this  technique. 


Requirements  should  be  thoroughly  re¬ 
viewed  to  identify  those  that  drive  perfor¬ 
mance.  This  will  require  the  "flow  down" 
of  performance  requirements  to  compo¬ 
nents  and  subassemblies  and  the  identifi¬ 
cation  of  technologies /techniques  to  be 
used  in  these  components/subassemblies 
that  may  significantly  affect  the  system's 
ability  to  meet  users'  needs. 

Designers  should  determine  the  sensitiv¬ 
ity  of  system  performance  to  the  require¬ 
ments  and  threat  and  identify  risk  drivers. 
Models  and  simulations  are  useful  tools  to 
determine  this  sensitivity.  For  example,  the 
U.S.  Army  Materiel  System  Analysis  Ac¬ 
tivity  (AMSAA)  has  such  an  analytic 
model,  the  AMSAA  Risk  Assessment  Meth¬ 
odology.  The  AMSAA  point  of  contact  can 
be  reached  at  telephone  number  (410)  278- 
6627.  PMWS  can  also  be  useful. 

The  risk  identified  in  this  technique  should 


54 


be  documented  in  a  program-specific  for¬ 
mat,  such  as  a  RIF. 

5.4.5  Cost  Risk  Assessment 

5.4.5.1  Description.  This  technique  pro¬ 
vides  a  program-level  cost  estimate  at 
completion  (E AC)  that  is  a  function  of  per¬ 
formance  and  schedule  risks.  It  uses  the 
results  of  previous  assessments  of  low-level 
WBS  elements  and  cost  probability  distri¬ 
butions  developed  for  each  of  the  elements. 
These  individual  WBS  elements  are  aggre¬ 
gated  using  a  Monte  Carlo  simulation  to 
obtain  a  probability  distribution  of  the  pro¬ 
gram-level  cost  EAC  probability  distribu¬ 
tion  function.  These  results  are  then  ana¬ 
lyzed  to  determine  the  actual  risk  of  cost 
overruns  and  to  identify  the  cost  drivers. 

The  use  of  these  lower  level  cost  probabil¬ 
ity  distributions  as  the  basis  for  the  pro- 
gram-level  cost  estimate  results  in  a  more 
realistic  EAC  than  the  commonly  used 
single  point  estimates  for  WBS  elements, 
since  they  address  both  the  probability  of 
occurrence  and  consequences  of  potential 
risk  events.  Their  use  also  eliminates  a 
major  cause  of  underestimating  (use  of 
point  estimates)  and  permits  the  identifi¬ 
cation  of  performance  or  schedule  causes 
of  cost  risk.  Thus,  this  technique  provides 
a  sound  basis  for  the  determination  of  an 
"acceptable"  level  of  cost  risk. 

This  technique  can  be  used  in  any  of  the 
acquisition  phases,  preferably  at  least  once 
per  phase  beginning  in  Phase  0;  however, 
for  contractors  using  an  earned  value  sys¬ 
tem,  it  will  be  done  as  part  of  the  normal 
management  of  a  program.  It  should  be 
used  in  conjunction  with  performance  and 
schedule  risk  assessments  and  may  be  per¬ 
formed  by  small  Government-industry 
teams  consisting  of  risk  analysts,  cost  ana¬ 
lysts,  schedule  analysts  and  technical  ex¬ 
perts  who  understand  the  significance  of 


previous  performance  and  schedule  risk 
assessments.  They  should  report  to  the  Pro¬ 
gram  IPT.  This  technique  requires  close 
and  continuous  cooperation  among  cost 
analysts  and  knowledgeable  technical  per¬ 
sonnel  and  the  support  of  the  prime 
contractor's  senior  management  to  help  get 
valid  cost  data. 

5.4.5.2  Procedures.  Figure  5-8  depicts  the 
process  used  in  applying  this  technique.  The 
first  step  is  to  identify  the  lowest  WBS  level 
for  which  cost  probability  distribution  func¬ 
tions  will  be  constructed.  The  level  selected 
will  depend  on  the  program  phase;  e.g.,  dur¬ 
ing  Phase  0,  it  may  not  be  possible  to  go  be¬ 
yond  level  2  or  3,  simply  because  the  WBS 
has  not  yet  been  developed  to  lower  levels. 
As  the  program  advances  into  subsequent 
phases  and  the  WBS  is  expanded,  it  will  be 
possible  and  necessary  to  go  to  lower  levels 
(4,  5,  or  lower).  Specific  performance  and 
schedule  risks  are  then  identified  for  these 
WBS  elements. 

To  develop  the  WBS  elements  cost  prob¬ 
ability  distribution  functions,  the  team, 
working  with  the  prime  contractor's  WBS 
element  managers,  determines  the  cost 
range  for  each  element  being  investigated. 
The  validity  of  the  cost  data  used  to  con¬ 
struct  the  distribution  is  critical.  In  fact, 
collecting  good  data  is  the  largest  part  of 
the  cost  risk  job.  Consequently,  PMOs 
should  place  major  emphasis  on  this  effort. 

The  element  cost  probability  distribution 
functions  are  then  aggregated  using  a 
Monte  Carlo  simulation  program.  All 
Monte  Carlo  processes  contain  limitations, 
but  they  are  more  nearly  accurate  than 
point  estimates.  Any  number  of  these 
simulations  are  readily  available  to  per¬ 
form  this  aggregation,  and  one  that  meets 
the  specific  needs  of  the  program  should 
be  selected.  The  results  of  this  step  will  be 
a  program-level  cost  EAC  and  a  cost  dis- 
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Figure  5-8.  Cost  Risk  Assessment  Top-Level  Diagram 


tribution  function  that  shows  the  cumula¬ 
tive  probability  associated  with  different 
cost  values.  These  outputs  are  then  ana¬ 
lyzed  to  determine  the  level  of  cost  risk  and 
to  identify  the  specific  cost  drivers.  Cost 
risk  is  determined  by  comparing  the  EAC 
with  the  cost  baseline  developed  as  part  of 
the  acquisition  program  baseline.  Since  the 
EAC  and  program  cost  distribution  are 
developed  from  WBS  element  risk  assess¬ 
ments,  it  is  possible  to  determine  the  cost 
risk  drivers.  The  cost  drivers  can  also  be 
related  back  to  the  appropriate  perfor¬ 
mance  and  schedule  risks.  The  results  of 
the  analysis  (cost  risks  and  drivers)  should 
be  documented  in  RIFs. 

5.4.6  Quantified  Schedule  Risk 
Assessment 

5.4.6.1  Description.  This  technique  provides 
a  means  to  determine  program-level  sched¬ 
ule  risk  as  a  function  of  risk  associated  with 


various  activities  that  comjxrse  the  program. 
It  estimates  the  program-level  schedule  by 
developing  probability  distributions  for  each 
activity  duration  and  aggregating  these  dis¬ 
tributions  using  a  Monte  Carlo  simulation  or 
otiher  analytical  tools.  The  resulting  program- 
level  schedule  is  then  analyzed  to  determine 
the  actual  schedule  risk  and  to  identify  the 
schedule  drivers. 

This  technique  expands  the  commonly 
used  Critical  Path  Method  (CPM)  of  devel¬ 
oping  a  program  schedule  to  obtain  a  real¬ 
istic  estimate  of  schedule  risk.  The  basic 
CPM  approach  uses  single  point  estimates 
for  the  duration  of  program  activities  to  de¬ 
velop  the  program's  expected  duration  and 
schedule.  It  invariably  leads  to  underesti¬ 
mating  the  time  required  to  complete  the 
program  and  schedule  overruns,  primarily 
because  the  point  estimates  do  not  ad¬ 
equately  address  the  uncertainty  inherent 
in  individual  activities.  The  uncertainty 
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can  be  caused  by  a  number  of  factors  and 
may  be  a  reflection  of  the  risk  present  in 
the  activity. 

The  quantified  schedule  technique  accounts 
for  uncertainty  by  using  a  range  of  time  that 
it  will  take  to  complete  each  activity  instead 
of  single  point  estimates.  These  ranges  are 
then  combined  to  determine  the  program- 
level  schedule  estimate.  This  approach  en¬ 
ables  PMs  to  estimate  early  in  a  program  if 
there  is  a  significant  likelihood  of  overrun¬ 
ning  the  program  schedule  and  by  how 
much.  It  ako  identifies  program  activities 
that  are  on  the  "highest  risk  path." 

Thk  technique  can  be  used  in  any  acquisi¬ 
tion  phase  beginning  with  the  completion  of 
the  fet  statement  of  work.  The  schedule 
probability  dktribution  function  for  each  key 
activity  should  be  developed  as  soon  as  the 
activity  is  included  in  the  master  schedule. 
The  dktribution  functions  should  be  periodi¬ 
cally  reviewed  and  revised,  if  necessary,  at 
least  once  per  phase.  The  technique  should 


be  applied  by  a  small  Government-industry 
team  consisting  of  schedule  analysts  and 
technical  experts  who  understand  the  signifi¬ 
cance  of  prior  risk  performance  assessments. 

5.4.6.2  Procedures.  Figure  5-9  shows  the 
process  used  in  thk  technique.  The  first 
step  is  to  identify  the  lowest  activity  level 
for  which  duration/ schedule  probability 
dktribution  functions  will  be  constructed. 
The  WBS  should  be  used  as  the  starting 
point  for  identifying  activities  and  con¬ 
structing  a  network  of  activities.  The  WBS 
level  selected  will  depend  on  the  program 
phase. 

Next,  the  contractor  should  construct  a 
CPM  schedule  for  these  activities.  To  de¬ 
velop  the  activity  duration  probability  dk¬ 
tribution  functions,  the  team,  working  with 
the  prime  contractor's  WBS  element  man¬ 
agers,  determines  and  analyzes  duration 
range  for  each  activity  being  investigated. 
This  analysis  should  be  done  by  schedule 
analysts  working  closely  with  knowledge- 
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Figure  5-9.  Schedule  Risk  Assessment  Technique  Inputs  and  Outputs 
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able  technical  people. 

The  activity  duration  probability  distribu¬ 
tion  functions  are  aggregated  using  a 
Monte  Carlo  simulation  program,  such  as 
©Risk,  Risk  +  for  Microsoft  Project,  or  Crys¬ 
tal  Ball.  The  result  of  this  step  is  a  pro¬ 
gram-level  schedule  and  distribution  func¬ 
tion  that  shows  the  cumulative  probabil¬ 
ity  associated  with  different  duration  val¬ 
ues.  These  outputs  are  then  analyzed  to 
determine  the  level  of  schedule  risk  and  to 
identify  the  specific  schedule  drivers.  Risk 
is  determined  by  comparing  the  program- 
level  schedule  with  the  schedule  baseline 
developed  as  part  of  the  acquisition  pro¬ 
gram  baseline.  The  fact  that  the  schedule 
and  distribution  are  developed  from  WBS 
element  risk  assessments  makes  it  possible 
to  determine  the  schedule  risk  drivers. 
These  drivers  can  also  be  related  back  to 
the  appropriate  performance  risks.  The 
results  of  the  analysis  (schedule  risks  and 
drivers)  should  be  documented  in  RIFs. 
The  analysis  requires  continued  close  co¬ 
operation  between  the  schedule  analysts 
and  technical  personnel  familiar  with  the 
details  of  the  program. 

5.4.7  Expert  Interviews 

5.4.7.1  Description.  A  difficult  part  of  the 
risk  management  process  is  data  gathering. 
This  technique  provides  a  means  for  col¬ 
lecting  risk-related  data  from  subject-mat¬ 
ter  experts  and  from  people  who  are  inti¬ 
mately  involved  with  the  various  aspects 
of  the  program.  It  relies  on  "expert"  judg¬ 
ment  to  identify  and  analyze  risk  events, 
develop  alternatives,  and  provide  "ana¬ 
lyzed"  data.  It  is  used  almost  exclusively 
in  a  support  role  to  help  develop  technical 
data,  such  as  probability  and  consequences 
information,  required  by  a  primary  risk  as¬ 
sessment  technique.  It  can  address  all  the 
functional  areas  that  make  up  the  critical 
risk  areas  and  processes,  and  can  be  used 


in  support  of  risk  handling. 

Expert  judgment  is  a  sound  and  practical 
way  of  obtaining  necessary  information 
that  is  not  available  elsewhere  or  practical 
to  develop  using  engineering  or  scientific 
techniques.  However,  interviewers  should 
be  aware  that  expert  opinions  may  be  bi¬ 
ased  because  of  over-reliance  on  certain  in¬ 
formation  and  neglect  of  other  information; 
unwarranted  confidence;  the  tendency  to 
recall  most  frequent  and  most  recent 
events;  a  tendency  to  neglect  rare  events; 
and  motivation.  Results  may  have  to  be 
tempered  because  of  these  biases. 

5.4.7.2  Procedures.  Figure  5-10  depicts  the 
process  used  in  this  technique.  The  first 
step  in  the  process  is  to  identify  risk  areas 
and  processes  that  are  to  be  evaluated  us¬ 
ing  the  expert  interview  technique.  Other 
techniques  described  in  this  section  (e.g., 
WBS  Risk  Assessment,  Process  Risk  Assess¬ 
ment,  etc.)  can  be  used  for  this  purpose. 

Once  the  areas  and  processes  are  known, 
subject-matter  experts  and  program/ contrac¬ 
tor  personnel  knowledgeable  of  the  areas  and 
processes  should  be  identified  to  be  inter¬ 
viewed.  Similarly,  qualified  interviewers 
should  be  selected  for  each  area  and  process. 

Interviewers  should  prepare  themselves  by 
preparing  a  strategy  and  selecting  a  meth¬ 
odology  for  analysis  and  quantification  of 
data.  TTie  references  list  sources  for  practical 
techniques  for  quantifying  expert  judgment. 

After  the  interview,  evaluators  analyze  the 
data  for  consistency,  resolve  any  issues,  and 
document  the  results.  Commercial 
"Groupware"  software  is  available  to  as¬ 
sist  in  compiling  and  documenting  the  re¬ 
sults  of  interviews. 

5.4.8  Analogy  Comparison/ 

Lessons-Learned  Studies 
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Figure  5-10.  Expert  Interview  Technique  Inputs  and  Outputs 


5.4.8.1  Description.  This  technique  uses  les¬ 
sons  learned  and  historical  information  about 
the  risk  associated  with  programs  that  are 
similar  to  the  new  system  to  identify  the  risk 
associated  with  a  new  program.  It  is  nor¬ 
mally  used  to  support  other  primary  risk  as¬ 
sessment  techniques,  e.g.,  Product  (WBS) 
Risk  Assessment,  Process  Risk  Assessment, 
etc.  The  technique  is  based  upon  the  con¬ 
cept  that  "new"  programs  are  originated  or 
evolved  from  existing  programs  or  simply 
repre^nt  a  new  combination  of  existing  com¬ 
ponents  or  subsystems.  A  logical  extension 
of  this  premise  is  that  key  insights  can  be 
gained  concerning  aspects  of  a  current 
program's  rislcs  by  examining  the  successes, 
failures,  problems,  and  solutions  of  similar 
existing  or  past  programs.  This  technique 
addresses  all  the  functional  areas  that  make 
up  the  critical  risk  areas  and  processes. 

5.4.8.2  Procedures.  Figure  5-11  depicts  the 
process  used  in  this  technique.  The  first 
step  in  this  approach  is  to  select  or  develop 
a  baseline  comparison  system  (BCS)  that 
closely  approximates  the  characteristics  of 


the  new  system/equipment  to  as  low  a 
level  as  possible  and  uses  the  processes 
similar  to  those  that  are  needed  to  develop 
the  new  system.  For  processes,  industry¬ 
wide  best  practices  should  be  used  as  a 
baseline.  The  PMWS  is  a  useful  tool  for 
identifying  these  best  practices. 

Relevant  BCS  data  are  then  collected,  ana¬ 
lyzed,  and  compared  with  the  new  system 
requirements.  The  BCS  data  may  require 
adjustment  to  make  a  valid  comparison;  for 
example,  apply  inflation  index  for  cost 
comparisons,  adjust  design  schedule  for 
software  evolution  versus  software  devel¬ 
opment,  etc.  The  comparisons  can  be  a 
major  source  of  risk  assessment  data  and 
provide  some  indication  of  areas  that 
should  be  investigated  further. 

5.5  RISK  PRIORITIZATION 
5.5.1  Description 

This  technique  provides  a  means  to  priori¬ 
tize  the  risl«  present  in  a  program.  The 
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Figure  5-11.  Analogy  Comparison/Lessons  Learned  Studies  Top-Level  Diagram 


prioritized  list  provides  the  basis  for  de¬ 
veloping  handling  plans,  preparing  a  han¬ 
dling  task  sequence  list,  and  allocating 
handling  resources. 

When  using  this  technique,  PMs  establish 
definitive  criteria  to  evaluate  the  risks,  such 
as,  probability  of  failure,  (Pp),  consequence 
of  failure  (Cp),  along  with  any  other  factors 
considered  appropriate.  The  risks  are  evalu¬ 
ated  using  quahtative  expert  judgment  and 
multi-voting  methods  to  prioritize  and  ag¬ 
gregate  risks.  (See  SEI,  Continuous  Risk  Man¬ 
agement,  1996,  for  a  discussion  of  multivoting 
methods.)  A  qualitative  approach  using  sub¬ 
ject  matter  experts  is  generally  preferred  in 
this  technique  because  of  the  tendency  to  rely 
on  ordinal  values  to  describe  Cp  and  the  in¬ 
herent  inaccuracies  resulting  from  any  at¬ 
tempts  to  use  quantifiable  methods  with  or¬ 
dinal  values. 


These  techniques  should  be  used  appropri¬ 
ately  during  Phases  0, 1,  and  H,  at  the  con¬ 
clusion  of  a  major  risk  assessment  under¬ 
taking,  when  there  has  been  a  significant 
change  in  the  acquisition  strategy,  when 
risk  monitoring  indicates  significant 
changes  in  the  status  of  a  number  of  risks, 
and  prior  to  a  milestone  review. 

The  PMO  risk  management  coordinator  (if 
assigned)  may  function  as  a  facilitator  and 
support  the  program  IPT  in  applying  these 
techniques. 

5.5.2  Procedures 

Figure  5-12  depicts  the  process  used  to  pri¬ 
oritize  the  risks  present  in  a  program.  The 
inputs  of  this  process  are  risks  that  have 
been  identified. 

The  evaluation  team,  through  consensus  or 
as  directed  by  the  Risk  Management  Plan, 
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Figure  5-12.  Risk  Prioritization  Technique  Inputs  and  Output 


selects  the  prioritization  criteria.  Pp  and  Cp 
should  always  be  part  of  the  criteria,  along 
with  any  other  appropriate  factors.  Ur¬ 
gency,  an  indication  of  the  time  available 
before  the  procedures  for  handling  the  spe¬ 
cific  risk  must  be  initiated,  is  often  consid¬ 
ered  in  the  evaluation.  The  PM  may  also 
choose  to  rank-order  the  prioritization  cri¬ 
teria,  e.g.,  consequence  is  more  important 
than  probability. 

A  multi-voting  method  is  useful  to  prioritize 
risks  (see  Scholtes,  1988;  Linstone,  1975).  The 
Delphi  method  is  a  simple  and  effective 
method  of  arriving  at  a  consensus  among  a 
group  of  experts.  The  procedure  is  for  team 
members  to  vote  on  the  priority  of  each  risk 
and  tally  the  results,  which  are  fed  back  to 
the  team.  Team  members  vote  again  and  the 
process  is  repeated  until  no  changes  occur  in 
the  results.  It  is  normal  to  reach  the  final 
outcome  within  a  few  voting  sessions.  If 
there  are  a  large  number  of  risks,  they  may 
be  broken  into  smaller  groups  for  ranking. 
As  a  general  rule,  no  more  than  10  items 
should  be  prioritized  per  vote.  The  results 
of  the  series  of  votes  are  documented  in  the 
risk  prioritization  list. 


PM  guidance,  which  operates  as  a  tech¬ 
nique  control  function,  can  be  used,  for 
example,  to  specify  prioritization  criteria 
and  prescribe  the  format  of  the  risk 
prioritization  list. 

5.5.2.1  Risk  Aggregation.  Figure  5-13 
shows  the  process  for  this  technique,  which 
relies  on  qualitative  judgment  and  multi¬ 
voting  methods  to  summarize  risks  at  the 
critical  risk  area  and  process  level  in  terms 
of  Pp  and  Cp.  The  risks  identified  in  the 
RIFs  and  Risk  Prioritization  List  are  first 
grouped  according  to  critical  risk  areas  and 
processes,  and  listed  in  priority  sequence. 

Within  each  area  and  process,  the  indi¬ 
vidual  risks  are  evaluated  agamst  a  set  of 
established  criteria  to  determine  the  over¬ 
all  aggregate  risk  rating  for  the  area/ pro¬ 
cess.  Aggregation  criteria  needs  to  be  es¬ 
tablished  separately  for  Pp  and  Cp ;  Ppand 
Cp  should  not  be  combined  into  a  single 
index,  e.g.,  moderate  risk.  Examples  of 
aggregation  criteria  include:  (1)  most  un¬ 
desirable  Ppand  Cp  of  all  the  risks  within  a 
risk  area  or  process  becomes  the  aggre¬ 
gated  values  for  the  area  or  process,  or  (2) 
the  Pp  and  Cp  for  each  area  or  process  rep- 
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Figure  5-13.  Risk  Aggregation  Technique  Inputs  and  Output 


resents  the  mean  value  for  that  area  or 
process. 

The  team  then  votes  on  each  risk  area  and 
process  to  determine  its  rating  for  Pp  and 
Cp,  and  the  results  are  documented.  In  ad¬ 
dition  to  the  Ppand  Cp  ratings  for  each  criti¬ 
cal  risk  area  and  process,  those  risks  that 
tend  to  "drive"  the  aggregate  risk  rating 
for  the  area /process  should  be  included  in 
a  list  of  aggregated  risks  to  give  substance 
to  the  aggregated  ratings,  e.g.,  all  risks  in 
which  either  PpOr  Cp  are  rated  as  high.  Fig¬ 
ure  5-14  provides  a  sample  list  of  aggre¬ 
gated  risks. 

Risk  Matrix  is  a  software  tool  that  is  de¬ 
signed  to  aid  in  managing  the  impacts  of 
key  risks  that  might  affect  a  project.  It  pro¬ 
vides  a  structured  method  for  prioritizing 
project  risks  and  for  tracking  the  status  and 
effects  of  risk-handling  efforts.  The  major 
feature  that  Risk  Matrix  offers  the  program 
office  is  a  means  to  rank  program  risks. 
This  is  helpful  in  differentiating  among 
risks  that  have  the  same  rating.  For  ex¬ 


ample,  if  a  program  has  eight  risks  that  the 
program  office  has  evaluated  as  high.  Risk 
Matrix  provides  the  means  to  rank  them 
in  order  of  severity.  The  user  can  use  this 
ranking  as  a  guide  to  help  focus  risk-han¬ 
dling  efforts.  Risk  Matrix  was  developed 
by  the  Air  Force  Electronic  Systems  Center 
(ESC)  and  The  Mitre  Corporation  and  is 
available  to  program  offices  free  of  charge. 

5.6  RISK-HANDLING  TECHNIQUES 
5.6.1  General 

After  the  program's  risks  have  been  as¬ 
sessed,  the  PM  must  develop  approaches 
to  handle  significant  ones  by  analyzing 
various  handling  techniques  and  selecting 
those  best  fitted  to  the  program's  circum¬ 
stances.  The  PM  should  reflect  these  ap¬ 
proaches  in  the  program's  acquisition  strat¬ 
egy  and  include  the  specifics  on  what  is  to 
be  done  to  deal  with  the  risk,  when  it 
should  be  accomplished,  who  is  respon¬ 
sible,  and  the  cost  and  schedule  impact. 

As  described  in  Chapter  2,  there  are  essen- 
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PROGRAM  XY  RISK  STATUS 


Risk  Area  Status:  Design 

Pp  iiL 

Cf  Hi 

Significant  Design  Risks: 

1.  Risk  Title:  Aircraft  Weight 

PfM 

Cp:  Mi 

Problem:  Exceed  aircraft  weight  budget  by  10%.  Decrease 
range-payload  by  4%. 

Action:  Developing  risk  handling  plan.  User  reviewing 
requirements. 


Risk  Area  Status:  Logistics  PpHi  Cp:  Mod/Hi 

Significant  Logistics  Risks:  None 


Figure  5-14.  Sample  of  a  List  of  Prioritized  Risks 


tially  four  risk-handling  techniques,  or 
options.  Risk  avoidance  eliminates  the 
sources  of  high  risk  and  replaces  them  with 
a  lower-risk  solution.  Risk  transfer  is  the 
reallocation  of  risk  from  one  part  of  the 
system  to  another,  or  the  reallocation  of 
risks  between  the  Government  and  the 
prime  contractor  or  within  Government 
agencies.  Risk  control  manages  the  risk  in  a 
manner  that  reduces  the  likelihood  of  its 
occurrence  and/or  minimizes  the  risk's 
effect  on  the  program.  Risk  assumption  is 
the  acknowledgment  of  the  existence  of  a 
particular  risk  situation  and  a  conscious 
decision  to  accept  the  associated  level  of 
risk  without  engaging  in  any  special  efforts 
to  control  it. 

In  determining  the  "best"  overall  risk-han¬ 
dling  strategy  and  specific  techniques  to 
be  adopted,  the  following  general  proce¬ 
dures  apply. 

For  each  identified  risk,  all  potentially  ap¬ 
plicable  techniques  should  be  identified  and 
evaluated,  using  the  following  criteria: 

•  Feasibility — ^Feasibility  is  the  ability 
to  implement  the  technique  and  includes 
an  evaluation  of  the  potential  impact  of  the 


technique  in  the  following  areas: 

-  Technical  considerations,  such  as 
testing,  manufacturing,  and  maintainabil¬ 
ity,  caused  by  design  changes  resulting 
from  risk-handling  techniques. 

-  Adequacy  of  budget  and  schedule 
flexibility  to  apply  the  technique. 

-  Operational  issues  such  as  usability 
(man-machine  interfaces),  transportability, 
and  mobility. 

-  Organizational  and  resource  consid¬ 
erations,  e.g.,  manpower,  training,  and  struc¬ 
ture. 

-  Environmental  issues,  such  as  the  use 
of  hazardous  materials  to  reduce  technical 
risk. 

-  External  considerations  beyond  the 
immediate  scope  of  the  program,  such  as 
the  impact  on  other  complementary  sys¬ 
tems  or  organizations. 

•  Cost  and  schedule  implications — ^The 
risk-handling  techniques  have  a  broad 
range  of  cost  implications  in  terms  of  dol¬ 
lars,  as  well  as  other  limited  resources,  e.g., 
critical  materials  and  national  test  facilities. 
The  magnitude  of  the  cost  and  schedule  im¬ 
plications  will  depend  on  circumstances 
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and  can  be  assessed  using  such  techniques 
as  cost-benefit  analyses  and  the  cost  and 
schedule  assessment  techniques  previously 
described.  The  approval  and  funding  of 
risk-handling  techniques  should  be  part  of 
the  tradeoff  process  that  estabUshes  and  re¬ 
fines  the  CAIV  cost  and  performance  goals. 

•  Effect  on  the  system's  technical  perfor¬ 
mance— The  risk-handling  techniques  may 
affect  the  system's  capability  to  achieve  the 
required  technical  performance  objectives. 
This  impact  must  be  clearly  understood  be¬ 
fore  adopting  a  specific  technique.  As  the 
risk-handling  techniques  are  assessed,  the 
PMO  should  attempt  to  identify  any  addi¬ 
tional  parameters  that  may  become  critical 
to  technical  performance  as  a  result  of  imple¬ 
menting  them.  Trade  studies  and  sensitivity 
analyses  can  be  useful  in  determining  the 
expected  effectiveness  of  this  approach. 

Once  the  risk-handling  technique  is  se¬ 
lected,  a  set  of  program  management  indi¬ 
cators  should  be  developed  to  provide 
feedback  on  program  progress,  effective¬ 
ness  of  the  risk-handling  options  selected, 
and  information  necessary  to  manage  the 
program.  These  indicators  should  consist 
of  cost  and  scheduling  data,  technical  per¬ 
formance  measures,  and  program  metrics. 

Subsequent  paragraphs  in  this  section  de¬ 
scribe  the  various  risk-handling  techniques 
cited  above. 

5.6.2  Risk  Avoidance 

5.6.2.1  Description.  This  technique  re¬ 
duces  risk  through  the  modification  or 
elimination  of  those  operational  require¬ 
ments  that  cause  the  risks.  It  requires  close 
coordination  with  the  users.  Since  this  tech¬ 
nique  results  in  the  reduction  of  risk,  it 
should  generally  be  initiated  in  the  devel¬ 
opment  of  a  risk-handling  plan.  It  can  be 
done  in  parallel  with  the  initial  operational 
requirements  analysis  and  should  be  sup¬ 


ported  by  a  cost-benefit  analysis. 

5.6.2.2  Procedures.  Analyzing  and  review¬ 
ing  the  proposed  system  in  detail  with  the 
user  is  essential  to  determine  the  drivers  for 
each  operational  requirement.  Operational 
requirements  scrubbing  involves  eliminating 
those  that  have  no  strong  basis.  This  also 
provides  the  PMO  and  the  user  with  an  im- 
derstanding  of  what  the  real  needs  are  and 
allows  them  to  establish  accurate  system  re¬ 
quirements  for  the  critical  performance. 
Operational  requirements  scrubbing  essen¬ 
tially  consists  of  developing  answers  to  the 
following  questions: 

•  Why  is  the  requirement  needed? 

•  What  will  the  requirement  provide? 

•  How  will  the  capability  be  used? 

•  Are  the  requirements  specified  in 
terms  of  functions  and  capabilities,  rather 
than  a  specific  design? 

Cost /requirement  trade  studies  are  used 
to  support  operational  requirements  scrub¬ 
bing.  These  trades  examine  each  require¬ 
ment  and  determine  the  cost  to  achieve 
various  levels  of  the  requirement  (e.g.,  dif¬ 
ferent  airspeeds,  range,  payloads).  The 
results  are  then  used  to  determine,  with  the 
user,  whether  a  particular  requirement 
level  is  worth  the  cost  of  achieving  that 
level.  Trade  studies  are  an  inherent  part  of 
the  systems  engineering  process.  (See 
Deskbook  2.6.1  for  details  on  systems  en¬ 
gineering  process.) 

5.6.3  Risk  Transfer 

5.6.3.1  Description.  This  technique  in¬ 
volves  the  reduction  of  risk  exposure  by 
the  reallocation  of  risk  from  one  part  of  the 
system  to  another  or  the  reallocation  of 
risks  between  the  Government  and  the 
prime  contractor. 

5.6.3.2  Procedures.  In  reallocating  risk. 
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design  requirements  that  are  risk  drivers 
are  transferred  to  other  system  elements, 
which  may  result  in  lower  system  risk  but 
still  meet  system  requirements.  For  ex¬ 
ample,  a  high  risk  caused  by  a  system  tim¬ 
ing  requirement  may  be  lowered  by  trans¬ 
ferring  that  requirement  from  a  software 
module  to  a  specially  designed  hardware 
module  capable  of  meeting  those  needs. 
The  effectiveness  of  requirements  realloca¬ 
tion  depends  on  good  system  engineering 
and  design  techniques.  In  fact,  efficient  al¬ 
location  of  those  requirements  that  are  risk 
drivers  is  an  integral  part  of  the  systems 
engineering  process.  Modularity  and  func¬ 
tional  partitioning  are  two  design  tech¬ 
niques  that  can  be  used  to  support  this  type 
of  risk  transfer.  In  some  cases,  this  ap¬ 
proach  may  be  used  to  concentrate  risk  ar¬ 
eas  in  one  area  of  the  system  design.  This 
allows  management  to  focus  attention  and 
resources  on  that  area. 

For  the  Government/ contractor  risk-trans¬ 
fer  approach  to  be  effective,  the  risks  trans¬ 
ferred  to  the  contractor  must  be  those  that 
the  contractor  has  the  capacity  to  control 
and  manage.  These  are  generally  risks  as¬ 
sociated  with  technologies  and  processes 
used  in  the  program — ^those  for  which  the 
contractor  can  implement  proactive  solu¬ 
tions.  The  types  of  risks  that  are  best  man¬ 
aged  by  the  Government  include  those  re¬ 
lated  to  the  stability  of  and  external  influ¬ 
ences  on  program  requirements,  funding, 
and  schedule,  for  example.  The  contractor 
can  support  the  management  of  these  risks 
through  the  development  of  flexible  pro¬ 
gram  plans,  and  the  incorporation  of  per¬ 
formance  margins  in  the  system  and  flex¬ 
ibility  in  the  schedule.  A  number  of  op¬ 
tions  are  available  to  implement  risk  trans¬ 
fer  from  the  Government  to  the  contractor: 
warranties,  cost  incentives,  product  perfor¬ 
mance  incentives,  and  various  types  of 
cost-based  contracts. 


5.6.4  Risk  Control 

5.6.4.1  Description.  In  this  risk-handling 
technique,  the  Government  and  contractor 
take  active  steps  to  reduce  the  likelihood 
of  a  risk  event  occurring  and  to  reduce  the 
potential  impact  on  the  program.  Most 
risk-control  steps  share  two  features:  they 
require  a  commitment  of  program  re¬ 
sources,  and  they  may  require  additional 
time  to  accomplish  them.  Thus,  the  selec¬ 
tion  of  risk-control  actions  will  undoubt¬ 
edly  require  some  tradeoff  between  re¬ 
sources  and  the  expected  benefit  of  the  ac¬ 
tions.  Some  of  the  many  risk-control  ac¬ 
tions  include  the  following: 

Multiple  Development  Efforts — ^The  use 
of  two  or  more  independent  design  teams 
(usually  two  separate  contractors,  although 
it  could  also  be  done  internally)  to  create  a 
system  that  meets  the  same  performance 
requirements. 

Alternative  Design — ^Sometimes,  a  design 
option  may  include  several  risky  approaches, 
of  which  one  or  more  must  come  to  fruition 
to  meet  S5^tem  requirements.  However,  if 
the  PMO  studies  the  risky  approaches,  it  may 
be  possible  to  discover  a  lower-risk  approach 
(with  a  lower  performance  capability).  These 
lower-risk  approaches  could  be  used  as  back¬ 
ups  for  those  cases  where  the  primary 
approach(es)  fail  to  mature  in  time.  This 
option  presumes  there  is  some  trading  room 
among  requirements.  Close  coordination 
between  the  developer  and  the  user  is  neces¬ 
sary  to  implement  lower  capability  options. 

Trade  Studies — ^Systems  engineering  de¬ 
cision  analysis  methods  include  trade  stud¬ 
ies  to  solve  a  complex  design  problem.  The 
purpose  of  the  trade  studies  is  to  integrate 
and  balance  all  engineering  requirements 
in  the  design  of  a  system.  A  properly  done 
trade  study  considers  risks  associated  with 
alternatives. 
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Early  Prototyping — ^The  nature  of  a  risk 
can  be  evaluated  by  a  prototype  of  a  sys¬ 
tem  (or  its  critical  elements)  built  and  tested 
early  in  the  system  development.  The  re¬ 
sults  of  the  prototype  can  be  factored  into 
the  design  and  manufacturing  process  re¬ 
quirements.  In  addition  to  full-up  systems, 
prototyping  is  very  useful  in  software  de¬ 
velopment  and  in  determining  a  system's 
man-machine  interface  needs.  The  key  to 
making  prototyping  successful  as  a  risk- 
control  tool  is  to  minimize  the  addition  of 
new  requirements  to  the  system  after  the 
prototype  has  been  tested  (i.e.,  requirement 
changes  not  derived  from  experience  with 
the  prototype).  Also,  the  temptation  to  use 
the  prototype  design  and  software  with¬ 
out  doing  the  necessary  follow-on  design 
and  coding/manufacturing  analyses 
should  be  avoided. 

Incremental  Development — ^Incremental 
development  is  completion  of  the  system 
design  and  deployment  in  steps,  relying  on 
pre-planned  product  improvements  (P3I) 
after  the  system  is  deployed  to  achieve  the 
final  system  capability.  Usually,  these 
added  capabilities  are  not  included  origi¬ 
nally  because  of  the  high  risk  that  they  will 
not  be  ready  along  with  the  remainder  of 
the  system.  Hence,  development  is  split, 
with  the  high-risk  portion  given  more  time 
to  mature.  The  basic  system,  however,  in¬ 
corporates  the  provisions  necessary  to  in¬ 
clude  the  add-on  capabilities.  In  P3I,  most 
of  the  system  requirements  are  achieved  by 
the  basic  system. 

Technology  Maturation  Efforts — ^Technol¬ 
ogy  maturation  is  an  off-line  development 
effort  to  bring  an  element  of  technology  to 
the  necessary  level  so  that  it  can  be  suc¬ 
cessfully  incorporated  into  the  system  (usu¬ 
ally  done  as  part  of  the  technology  transi¬ 
tion  process).  Normally,  technology  matu¬ 
ration  is  used  when  the  desired  technol¬ 
ogy  will  replace  an  existing  technology. 


which  is  available  for  use  in  the  system.  In 
those  cases,  technology  maturation  efforts 
are  used  in  conjunction  with  P3I  efforts. 
However,  it  can  also  be  used  when  a  criti¬ 
cal,  but  immature,  technology  is  needed. 
In  addition  to  dedicated  efforts  conducted 
by  the  PMO,  Service  or  DoD-wide  technol¬ 
ogy  improvement  programs  and  advanced 
technology  demonstrations  by  Govern¬ 
ment  laboratories  as  well  as  industry 
should  be  considered. 

Test-Analyze-And-Fix  (TAAF) — TAAF  is 
the  use  of  a  period  of  dedicated  testing  to 
identify  and  correct  deficiencies  in  a  design. 
It  was  originally  conceived  as  an  approach 
to  improve  reliability;  it  can  also  be  used  for 
any  system  parameter  whose  development 
could  benefit  from  a  dedicated  period  of  test¬ 
ing  and  analysis.  Although  a  valuable  aid  in 
the  development  process,  TAAF  should  not 
be  used  in  lieu  of  a  sound  design  process. 

Robust  Design — This  approach  uses  ad¬ 
vanced  design  and  manufacturing  tech¬ 
niques  that  promote  achieving  quality 
through  design.  It  normally  results  in 
products  with  little  sensitivity  to  variations 
in  the  manufacturing  process. 

Reviews,  Walkthroughs,  and  Inspections 
— ^These  three  risk  control  actions  can  be 
used  to  reduce  the  likelihood  and  poten¬ 
tial  consequences  of  risks  through  timely 
assessments  of  actual  or  planned  events  in 
the  development  of  the  product.  They  vary 
in  the  degree  of  formality,  level  of  partici¬ 
pants,  and  timing. 

Reviews  are  formal  sessions  held  to  assess 
the  status  of  the  program,  the  adequacy  and 
sufficiency  of  completed  events,  and  the  in¬ 
tentions  and  consistency  of  future  events. 
Reviews  are  usually  held  at  the  completion 
of  a  program  phase,  when  significant  prod¬ 
ucts  are  available.  The  team  conducting 
the  review  should  have  a  set  of  objectives 
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and  specific  issues  to  be  addressed.  The 
results  should  be  documented  in  the  form 
of  action  items  to  be  implemented  by  the 
PMO  or  contractor.  The  type  of  review  will 
dictate  the  composition  of  the  review  team, 
which  may  include  developers,  users,  man¬ 
agers,  and  outside  experts. 

A  walkthrough  is  a  technique  that  can  be  very 
useful  in  assessing  the  progress  in  the  devel¬ 
opment  of  high  or  moderate  risk  compo¬ 
nents,  especially  software  modules.  It  is  less 
formal  than  a  review,  but  no  less  rigorous. 
The  person  responsible  for  the  development 
of  the  component  "walks  through"  the  prod¬ 
uct  development  (to  include  perceptions  of 
what  is  to  be  done,  how  it  will  be  accom¬ 
plished,  and  the  schedule)  with  a  team  of 
subject-matter  experts.  The  team  reviews  and 
evaluates  the  progress  and  plans  for  devel¬ 
oping  the  product  and  provides  immediate 
and  less  formal  feedback  to  the  responsible 
person,  thus  enabling  improvements  or  cor¬ 
rective  actions  to  be  made  while  the  product 
is  still  imder  development.  This  technique 
is  applied  during  the  development  phases, 
as  opposed  to  reviews,  which  are  normally 
held  at  the  completion  of  a  phase  or  product. 

Inspections  are  conducted  to  evaluate  the 


correctness  of  the  product  under  develop¬ 
ment  in  terms  of  its  design,  implementa¬ 
tion,  test  plans,  and  test  results.  They  are 
more  formal  and  rigorous  than  either  re¬ 
views  or  walkthroughs  and  are  conducted 
by  a  team  of  experts  following  a  very  fo¬ 
cused  set  of  questions  concerning  all  as¬ 
pects  of  the  product. 

Design  of  Experiments — ^This  is  an  engi¬ 
neering  tool  that  identifies  critical  design 
factors  that  are  difficult  to  meet. 

Demonstration  Events — Demonstration 
events  are  points  in  the  program  (usually 
tests)  that  are  used  to  determine  if  risks  are 
being  successfully  abated.  Careful  review 
of  the  planned  development  of  each  risk 
area  will  reveal  a  number  of  opportunities 
to  verify  the  effectiveness  of  the  develop¬ 
ment  approach.  By  including  a  sequence 
of  demonstration  events  throughout  the  de¬ 
velopment,  PMO  and  contractor  personnel 
can  monitor  the  process  and  identify  when 
additional  efforts  are  needed.  Demonstra¬ 
tion  events  can  also  be  used  as  informa¬ 
tion-gathering  actions,  as  discussed  before, 
and  as  part  of  the  risk-monitoring  process. 
Table  5-2  contains  examples  of  demonstra¬ 
tion  events. 


Table  5-2.  Examples  of  Demonstration  Events 


ITEM 

DEMONSTRATION  EVENT 

COMPLETION  DATE 

Rocket  Motor 

Three  Case  Burst  Tests 

Propellant  Characterization 

Thermal  Barrier  Bond  Tests 

Ignition  and  Safe/Arm  Tests 

Nozzle  Assembly  Tests 

By  completion  of  preliminary 
design 

10  Development  Motor  Firings 

—  Temperature  and  Altitude  Cycle 

—  Vibration  and  Shock 

—  Aging 

By  completion  of  final  design 

Central  Computer 

Test  Breadboard 

Develop/Test  Unique  Microcircuits 

By  completion  of  preliminary 
design 

Build/Test  Prototype 

By  completion  of  final  design 

67 


Open  Systems — This  approach  involves 
the  use  of  widely  accepted  commercial 
specifications  and  standards  for  selected 
system  interfaces,  products,  practices,  and 
tools.  It  provides  the  basis  for  reduced  life- 
cycle  costs,  improved  performance,  and  en¬ 
hanced  interoperabihty,  especially  for  long¬ 
life  systems  with  short-life  technologies. 
Properly  selected  and  applied  commercial 
specifications  and  standards  can  result  in 
lower  risk  through  increased  design  flex¬ 
ibility;  reduced  design  time;  more  predict¬ 
able  performance;  and  easier  product  in¬ 
tegration,  support,  and  upgrade.  However, 
a  number  of  challenges  and  risks  are  asso¬ 
ciated  with  the  use  of  the  open  systems  ap¬ 
proach  and  must  be  considered  before 
implementation.  These  include  such  issues 
as:  maturity  and  acceptability  of  the  stan¬ 
dard,  and  its  adequacy  for  military  use;  the 
loss  of  control  over  the  development  of 
products  used  in  the  system;  the  amount 
of  product  testing  done  to  ensure  conform¬ 
ance  to  standards;  and  the  higher  configu¬ 
ration  management  workload  required. 

See  Deskbook  Section  1.2.2.2.5  for  a  more 
detailed  discussion  of  the  use  of  open  sys¬ 
tems.  Additional  information  is  also  avail¬ 
able  at  the  Open  Systems  Joint  Task  Force 
Website  at  www.acq.osd.mil/ osjtf/ 

Use  of  Standard  Items/Software  Reuse — 
The  use  of  standard  items  and  software  mod¬ 
ule  reuse  should  be  emphasized  to  the  ex¬ 
tent  possible  to  minimize  development  risk. 
Standard  items  range  from  components  and 
assemblies  to  full-up  systems.  A  careful  ex¬ 
amination  of  the  proposed  system  option  will 
often  find  more  opportunities  for  the  use  of 
standard  items  or  existing  software  modules 
than  first  considered.  Even  when  the  sys¬ 
tem  must  achieve  previously  unprecedented 
requirements,  standard  items  can  find  uses. 
Astiong  program  policy  emphasizing  the  use 
of  standard  items  and  software  reuse  is  of¬ 
ten  the  key  to  taking  advantage  of  this  source 


of  risk  control.  Standard  items  and  software 
modules  have  proven  characteristics  that  can 
reduce  risk.  However,  the  PMO  must  be  cau¬ 
tious  when  using  standard  items  in  environ¬ 
ments  and  applications  for  which  they  were 
not  designed.  A  misapplied  standard  item 
often  leads  to  failure. 

Two-Phase  EMD — This  risk-control  ap¬ 
proach  incorporates  a  formal  risk-reduc¬ 
tion  effort  in  the  initial  part  of  the  EMD 
phase.  It  may  involve  using  two  or  more 
contractors  with  a  down-select  occurring 
at  a  predefined  time  (normally  after  the 
preliminary  design  review).  A  logical  ex¬ 
tension  of  this  concept  is  the  "spiral"  de¬ 
velopment  model,  which  emphasizes  the 
evaluation  of  alternatives  and  risk  assess¬ 
ments  throughout  the  system's  develop¬ 
ment  and  initial  fielding. 

Use  of  Mockups— The  use  of  mockups,  es¬ 
pecially  man-machine  interface  mock-ups, 
can  be  used  to  conduct  early  exploration 
of  design  options.  They  can  assist  in  re¬ 
solving  design  uncertainties  and  provid¬ 
ing  users  with  early  views  of  the  final  sys¬ 
tem  configuration. 

Modeling/Simulation —  The  use  of  mod¬ 
eling  and  simulation  can  provide  insights 
into  a  system's  performance  and  effective¬ 
ness  sensitivities.  Decision  makers  can  use 
performance  predictions  to  assess  a 
system's  miUtary  worth  not  only  before  any 
physical  prototypes  are  built,  but  also 
throughout  the  system  life  cycle. 

Modeling  and  simulation  can  help  manage 
risk  by  providing  information  on  design 
capabilities  and  failure  modes  during  the 
early  stages  of  design.  This  allows  initial 
design  concepts  to  be  iterated  without  hav¬ 
ing  to  build  hardware  for  testing.  The  T&E 
community  can  use  predictive  simulations 
to  focus  the  use  of  valuable  test  assets  on 
critical  test  issues.  They  can  also  use  ex- 
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trapolated  simulations  to  expand  the  scope 
of  evaluation  into  areas  not  readily  testable, 
thus  reducing  the  risk  of  having  the  sys¬ 
tem  fail  in  the  outer  edges  of  the  "test  en¬ 
velope."  Additionally,  a  model  can  serve 
as  a  framework  to  bridge  the  missing  pieces 
of  a  complete  system  until  those  pieces 
become  available. 

Although  modeling  and  simulation  can  be 
a  very  effective  risk-handling  tool,  it  re¬ 
quires  resources,  commitment  to  refine 
models  as  the  system  under  development 
matures,  and  a  concerted  verification  and 
validation  effort  to  ensure  that  decisions  are 
based  on  credible  information. 

Key  Parameter  Control  Boards — ^When  a 
particular  parameter  (such  as  system 
weight)  is  crucial  to  achieving  the  overall 
program  requirements,  a  control  board  for 
that  parameter  may  be  appropriate.  This 
board  has  representatives  from  all  affected 
technical  functions  and  may  be  chaired  by 
the  PM.  It  provides  management  focus  on 
the  parameter  and  signals  the  importance 
of  achieving  the  parameter  to  the  technical 
community.  If  staffed  properly  by  all  af¬ 
fected  disciplines,  it  can  also  help  avoid 
sacrificing  other  program  requirements  to 
achieve  that  requirement. 

Process  Proofing — ^When  particular  pro¬ 
cesses,  especially  those  of  manufacturing 
and  support,  are  critical  to  achieving  sys¬ 
tem  requirements,  an  early  process  proof 
demonstration  is  useful  to  abate  risk.  If 
the  initial  proof  is  unsuccessful,  time  is  still 
available  to  identify  and  correct  deficien¬ 
cies  or  to  select  an  alternative  approach. 

Manufacturing  Screening — ^For  programs  in 
engineering,  manufacture,  and  development 
(EMD),  various  manufacturing  screens  (in¬ 
cluding  environmental  stress  screening)  can 
be  incorporated  into  test  article  production 
and  low-rate  initial  production  to  identify  de¬ 


ficient  manufacturing  processes.  These  data 
can  then  be  used  to  develop  the  appropriate 
corrective  actions. 

5.6.4.2  Procedures.  Risk  control  involves 
developing  a  risk-reduction  plan,  with  ac¬ 
tions  identified,  resourced,  and  scheduled. 
Success  criteria  for  each  of  the  risk-reduc¬ 
tion  events  should  also  be  identified.  The 
effectiveness  of  these  actions  must  be  moni¬ 
tored  using  the  types  of  techniques  de¬ 
scribed  in  Section  5.7. 

5.6.5  Risk  Assumption 

5.6.5.1  Description.  This  technique  is  used 
in  every  program  and  acknowledges  the  fact 
that,  in  any  program,  risks  exist  that  will  have 
to  be  accepted  without  any  special  effort  to 
control  them.  Such  risks  may  be  either  in¬ 
herent  in  the  program  or  may  result  from 
other  risk-controlling  actions  (residual  risks). 
The  fact  that  risks  are  assumed  does  not  mean 
that  they  are  ignored.  In  fact,  every  effort 
should  be  made  to  identify  and  rmderstand 
them  so  that  appropriate  management  action 
can  be  planned.  Ako,  risks  that  are  assumed 
should  be  monitored  during  development; 
this  monitoring  should  be  well-plarm^  from 
the  beginning. 

5.6.5.2  Procedures.  In  addition  to  the  iden¬ 
tification  of  risks  to  be  assumed,  the  fol¬ 
lowing  steps  are  key  to  successful  risk  as¬ 
sumption: 

•  Identify  the  resources  (time,  money, 
people,  etc.)  needed  to  overcome  a  risk  if  it 
materializes.  This  includes  identifying  the 
specific  management  actions  that  will  be 
used,  for  example,  redesign,  retesting,  re¬ 
quirements  review,  etc. 

•  Ensure  that  the  necessary  administra¬ 
tive  actions  are  taken  to  quicldy  implement 
these  management  actions,  such  as  con¬ 
tracts  for  industry  expert  consultants,  ar¬ 
rangements  for  test  facilities,  etc. 


69 


Whenever  a  risk  is  assumed,  a  schedule  and 
cost  reserve  should  be  set  aside  to  cover  the 
specific  actions  to  be  taken  if  the  risk  occurs. 
If  this  is  not  possible,  the  program  may  pro¬ 
ceed  within  the  funds  and  schedule  allotted 
to  the  effort.  If  the  program  cannot  achieve 
its  objectives,  a  decision  must  be  made  to  al¬ 
locate  additional  resources,  accept  a  lower 
level  of  capability  (lower  the  requirements), 
or  cancel  the  effort. 

5.7  RISK  MONITORING 
5.7.1  General 

Risk  monitoring  is  a  continuous  process  to 
systematically  track  and  evaluate  the  per¬ 
formance  of  risk-handling  actions  against 
estabhshed  metrics  throughout  the  acqui¬ 
sition  process.  It  should  also  include  re¬ 
sults  of  periodic  reassessments  of  program 
risk  to  evaluate  both  known  and  new  risks 
to  the  program.  If  necessary,  the  PMO 
should  reexamine  the  risk-handling  ap¬ 
proaches  for  effectiveness  while  conduct¬ 
ing  assessments.  As  the  program 
progresses,  the  monitoring  process  will 
identify  the  need  for  additional  risk-han¬ 
dling  options. 

An  effective  monitoring  effort  provides 
information  to  show  if  handling  actions  are 
not  working  and  which  risks  are  on  their 
way  to  becoming  actual  problems.  The 
information  should  be  available  in  suffi¬ 
cient  time  for  the  PMO  to  take  corrective 
action.  The  fimctioning  of  IPTs  is  crucial 
to  effective  risk  monitoring.  They  are  the 
"front  line"  for  obtaining  indications  that 
handling  efforts  are  achieving  their  desired 
effects. 

The  establishment  of  a  management  indi¬ 
cator  system  that  provides  accurate,  timely, 
and  relevant  risk  information  in  a  clear, 
easily  understood  manner  is  key  to  risk 
monitoring.  Early  in  the  planning  phase  of 


the  process,  PMOs  should  identify  specific 
indicators  to  be  monitored  and  information 
to  be  collected,  compiled,  and  reported. 
Normally,  documentation  and  reporting 
procedures  are  developed  as  part  of  risk 
management  planning  before  contract 
award  and  should  use,  as  much  as  possible, 
the  contractor's  reporting  system.  Specific 
procedures  and  details  for  risk  reporting 
should  be  included  in  the  risk  management 
plans  prepared  by  the  Government  and  the 
contractor. 

To  ensure  that  significant  risks  are  effec¬ 
tively  monitored,  handling  actions  (which 
include  specific  events,  schedules,  and 
"success"  criteria)  developed  during  pre¬ 
vious  risk  management  phases  should  be 
reflected  in  integrated  program  planning 
and  scheduling.  Identifying  these  han¬ 
dling  actions  and  events  in  the  context  of 
WBS  elements  establishes  a  linkage  be¬ 
tween  them  and  specific  work  packages, 
making  it  easier  to  determine  the  impact 
of  actions  on  cost,  schedule,  and  perfor¬ 
mance.  The  detailed  information  on  risk¬ 
handling  actions  and  events  should  be  con¬ 
tained  in  various  risk  management  docu¬ 
mentation  (both  formal  and  informal). 
Experience  has  shown  that  the  use  of  an 
electronic  online  database  that  stores  and 
permits  retrieval  of  risk-related  informa¬ 
tion  is  almost  essential  to  effective  risk 
monitoring.  The  database  selected  or  de¬ 
veloped  will  depend  on  the  program.  A 
discussion  of  risk  management  information 
systems  and  databases  and  suggested  data 
elements  to  be  included  in  the  databases  is 
contained  later  in  this  chapter. 

Many  techniques  and  tools  are  available  for 
monitoring  the  effectiveness  of  risk-han¬ 
dling  actions,  and  PMO  personnel  should 
select  those  that  best  suit  their  needs.  No 
single  technique  or  tool  is  capable  of  pro¬ 
viding  a  complete  answer — a  combination 
must  be  used.  In  general,  risk  monitoring 
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techniques  are  applied  to  follow  through 
on  the  planned  actions  of  the  risk-handling 
program.  They  track  and  evaluate  the  ef¬ 
fectiveness  of  handling  activities  by  com¬ 
paring  planned  actions  with  what  is  actu¬ 
ally  achieved.  These  comparisons  may  be 
as  straightforward  as  actual  versus  planned 
completion  dates,  or  as  complex  as  detailed 
analysis  of  observed  data  versus  planned 
profiles.  In  any  case,  the  differences  be¬ 
tween  planned  and  actual  data  are  exam¬ 
ined  to  determine  status  and  the  need  for 
any  changes  in  the  risk-handling  approach. 

PMO  personnel  should  also  ensure  that  the 
indicators/metrics  selected  to  monitor  pro¬ 
gram  status  adequately  portray  the  true 
state  of  the  risk  events  and  handUng' ac¬ 
tions.  Otherwise,  indicators  of  risks  that 
are  about  to  become  problems  will  go  un¬ 
detected.  Subsequent  sections  identify  spe¬ 
cific  techniques  and  tools  that  will  be  use¬ 
ful  to  PMOs  in  monitoring  risks  and  pro¬ 
vide  information  on  selecting  metrics  that 
are  essential  to  the  monitoring  effort.  The 
techniques  focus  primarily  at  the  program 
level,  addressing  cost,  schedule,  and  per¬ 
formance  risks. 

5.7.2  Earned  Value  Management 

5.7.2.1  Description.  Earned  value  (EV)  is 
a  management  technique  that  relates  re¬ 
source  planning  to  schedules  and  to  tech¬ 
nical  performance  requirements.  It  is  use¬ 
ful  in  monitoring  the  effectiveness  of  risk¬ 
handling  actions  in  that  it  provides  peri¬ 
odic  comparisons  of  the  actual  work  ac¬ 
complished  in  terms  of  cost  and  schedule 
with  the  work  planned  and  budgeted. 
These  comparisons  are  made  using  a  per¬ 
formance  baseline  that  is  established  by  the 
contractor  and  the  PM  at  the  begiiming  of 
the  contract  period.  This  is  accomplished 
through  the  Integrated  Baseline  Review 
(IBR)  process.  The  baseline  must  capture 
the  entire  technical  scope  of  the  program 


in  detailed  work  packages.  The  baseline 
also  includes  the  schedule  to  meet  the  re¬ 
quirements  as  well  as  the  resources  to  be 
applied  to  each  work  package.  Specific 
risk-handling  actions  should  be  included 
in  these  packages.  See  Deskbook  Section 
2.B.2.1  for  a  more  detailed  discussion  of 
Earned  Value  and  IBR. 

5.7.2.2  Procedures.  The  periodic  EV  data 
can  provide  indications  of  risk  and  the  ef¬ 
fectiveness  of  handling  actions.  When  vari¬ 
ances  in  cost  or  schedule  begin  to  appear 
in  the  work  packages  containing  risk-han¬ 
dling  actions,  the  appropriate  IPTs  can  ana¬ 
lyze  the  data  to  isolate  causes  of  the  vari¬ 
ances  and  gain  insights  into  the  need  to 
modify  handling  actions. 

5.7.3  Technical  Performance 
Measurement 

5.73.1  Description.  Technical  performance 
measurement  (TPM)  is  a  technique  that  com¬ 
pares  estimated  values  of  essential  technical 
performance  parameters  with  achieved  val¬ 
ues,  and  determines  the  impact  of  any  dif¬ 
ferences  on  system  effectiveness.  This  tech¬ 
nique  can  be  useful  in  risk  monitoring  by 
comparing  plaimed  and  achieved  values  of 
parameters  in  areas  of  known  risk.  The  pe¬ 
riodic  application  of  this  technique  can  pro¬ 
vide  early  and  continuing  predictions  of  the 
effectiveness  of  risk-handling  actions  or  the 
detection  of  new  risks  before  irrevocable 
impacts  on  the  cost  or  schedule  occur. 

5.7.3.2  Procedures.  The  technical  perfor¬ 
mance  parameters  selected  should  be  those 
that  are  indicators  of  progress  in  the  risk¬ 
handling  action  employed.  They  can  be 
related  to  system  hardware,  software,  hu¬ 
man  factors,  and  logistics — ^any  product  or 
functional  area  of  the  system.  Parameter 
values  to  be  achieved  through  the  planned 
handling  action  are  forecast  in  the  form  of 
planned  performance  profiles.  Achieved 
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values  for  these  parameters  are  compared 
with  the  expected  values  from  the  profile, 
and  any  differences  are  analyzed  to  get  an 
indication  of  the  effectiveness  of  the  han¬ 
dling  action.  For  example,  suppose  a  sys¬ 
tem  requires  the  use  of  a  specific  technol¬ 
ogy  that  is  not  yet  mature  and  the  use  of 
which  has  been  assessed  as  high  risk.  The 
handling  technique  selected  is  risk  control, 
and  an  off-line  technology  maturation  ef¬ 
fort  will  be  used  to  get  the  technology  to 
the  level  where  the  risk  is  acceptable.  The 
technology  is  analyzed  to  identify  those  pa¬ 
rameters  that  are  key  drivers,  and  perfor¬ 
mance  profiles  that  will  result  from  a  suf¬ 
ficiently  mature  technology  are  established. 
As  the  maturation  effort  progresses,  the 
achieved  values  of  these  parameters  are 
compared  with  the  planned  profile.  If  the 
achieved  values  meet  the  planned  profile, 
it  is  an  indicator  that  the  risk-handling  ap¬ 
proach  is  progressing  satisfactorily;  if  the 
achieved  values  fall  short  of  the  expected 
values,  it  is  an  indicator  that  the  approach 
is  failing  to  meet  expectations. 

5.7.4  Integrated  Planning  and 
Scheduling 

5.7.4.1  Description.  Once  a  contract  has 
been  awarded,  techniques  such  as  inte¬ 
grated  planning  and  scheduling  (inte¬ 
grated  master  plans  and  integrated  master 
schedules)  can  become  invaluable  program 
baseline  and  risk-monitoring  tools.  Inte¬ 
grated  planning  identifies  key  events,  mile¬ 
stones,  reviews,  all  integrated  technical 
tasks,  and  risk-reduction  actions  for  the 
program,  along  with  accomplishment  cri¬ 
teria  to  provide  a  definitive  measure  that 
the  required  maturity  or  progress  has  been 
achieved.  Integrated  scheduling  describes 
the  detailed  tasks  that  support  the  signifi¬ 
cant  activities  identified  in  integrated  plan¬ 
ning  and  timing  of  tasks.  Also,  the  inte¬ 
grated  schedule  can  include  the  resources 
planned  to  complete  the  tasks.  The  events. 


tasks,  and  schedule  resulting  from  inte¬ 
grated  planning  are  linked  with  contract 
specification  requirements,  WBS,  and  other 
techniques  such  as  TPM.  When  the  events 
and  tasks  are  related  to  risk-reduction  ac¬ 
tions,  this  linkage  provides  a  significant 
monitoring  tool,  giving  specific  insights 
into  the  relationships  among  cost,  sched¬ 
ule,  and  performance  risks. 

5.7.4.2  Procedures.  In  integrated  planning, 
the  Government  and  contractor  (or  other 
performing  activity)  should  identify  key 
activities  of  the  program,  to  include  risk¬ 
handling  actions  and  success  criteria.  The 
contractor  should  then  prepare  the  inte¬ 
grated  schedule  reflecting  the  planned 
completion  of  tasks  associated  with  these 
activities.  As  the  program  progresses,  the 
PMO  can  monitor  effectiveness  of  handling 
activities  included  in  the  integrated  plan¬ 
ning  events  and  schedule  by  comparing 
observed  activity  results  with  their  crite¬ 
ria  and  determining  any  deviations  from 
the  planned  schedule.  Any  failures  of  han¬ 
dling  actions  to  meet  either  the  event  crite¬ 
ria  or  schedule  should  be  analyzed  to  de¬ 
termine  the  deviation's  impact,  causes,  and 
need  for  any  modifications  to  the  risk-han¬ 
dling  approach. 

5.7.5  Watchlist 

5.7.5.1  Description.  The  watchlist  is  a  list¬ 
ing  of  critical  areas  to  which  management 
should  pay  special  attention  during  pro¬ 
gram  execution.  It  is  a  straightforward, 
easily  prepared  document  that  can  range 
in  complexity  from  a  simple  list  of  the  iden¬ 
tified  risks  to  one  that  includes  such  things 
as  the  priority  of  the  risk,  how  long  it  has 
been  on  the  watchlist,  the  handling  actions, 
planned  and  actual  completion  dates  for 
handling  actions,  and  explanations  for  any 
differences.  See  Table  5-3  for  an  example 
watchlist. 
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Table  5-3.  Watchlist  Example 


Potential  Risk 

Risk  Reduction  Actions 

Action 

Due  Date 

Date 

Explanation 

Area 

Code 

Completed 

•  Accurately 

•  Use  multiple  finite 

SEA 

31  Aug  97 

predicting 

element  codes  & 

03P31 

shock 

simplified  numerical 

environment 

models  for  early 

shipboard 

assessments. 

equipment  will 

•  Shock  test  simple  isolated 

SEA 

31  Aug  98 

experience. 

structure,  simple  isolated 
deck,  and  proposed 
isolated  structure  to 
improve  confidence  in 
predictions. 

03P31 

•  Evaluating 

•  Concentrate  on  acoustic 

SEA 

31  Aug  97 

acoustic 

modeling  and  scale 

03TC 

impact  of  ship 

testing  of  technologies  not 

systems  that 

demonstrated 

are  not  similar 

successfully  in  large  scale 

to  previous 

tests  or  full  scale  trials. 

designs. 

•  Factor  acoustic  signature 

SEA 

31  Aug  98 

mitigation  from  isolated 
modular  decks  into 

03TC 

system  requirements. 
Continue  model  tests  to 
validate  predictions  for 
isolated  decks. 

5.7.5^  Procedures.  Watchlist  development 
is  based  on  the  results  of  the  risk  assessment. 
It  is  conunon  to  keep  the  number  of  risks  on 
the  watchlist  relatively  small,  focusing  on 
those  that  can  have  the  greatest  impact  on 
the  program.  Items  can  be  added  as  the  pro¬ 
gram  unfolds  and  periodic  reassessments  are 
conducted.  If  a  considerable  number  of  new 
risks  are  significant  enough  to  be  added  to 
the  watchlist,  it  may  be  an  indicator  that  the 
original  assessment  was  not  accurate  and  that 
program  risk  is  greater  than  initially  thought. 
It  may  also  indicate  that  the  program  is  on 
the  verge  of  becoming  out  of  control.  If  a 
risk  has  been  on  the  watchlist  for  a  long  time 
because  of  a  lack  of  risk-handling  progress, 
a  reassessment  of  the  risk  or  the  handling 
approach  may  be  necessary.  Items  on  the 


watchlist  should  be  reviewed  during  the  vari¬ 
ous  program  reviews/ meetings,  both  formal 
and  informal. 

5.7.6  Reports 

5.7.6.1  Description.  Reports  are  used  to  con¬ 
vey  information  to  decision  makers  and  pro¬ 
gram  team  members  on  the  status  of  risks 
and  the  effectiveness  of  risk-handling  actions. 
Risk-related  reports  can  be  presented  in  a 
variety  of  ways,  ranging  from  informal  ver¬ 
bal  reports  when  time  is  of  the  essence  to  for¬ 
mal  summary-type  reports  presented  at  mile¬ 
stone  reviews.  The  level  of  detail  presented 
win  depend  on  the  audience. 

5.7.6.2  Procedures.  Successful  risk  man¬ 
agement  programs  include  timely  report- 
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ing  of  results  of  the  monitoring  process. 
Reporting  requirements  and  procedures,  to 
include  format  and  frequency,  are  normally 
developed  as  part  of  risk  management 
planning  and  are  documented  in  the  risk 
management  plan.  Reports  are  normally 
prepared  and  presented  as  part  of  routine 
program  management  activities.  They  can 
be  effectively  incorporated  into  program 
management  reviews  and  technical  mile¬ 
stones  to  indicate  any  technical,  schedule, 
and  cost  barriers  to  the  program  objectives 
and  milestones  being  met.  One  example 
of  a  status  presentation  is  shown  in  Figure 
5-15.  It  shows  top-level  risk  information 
that  can  be  useful  to  the  PMO  as  well  as 
others  external  to  the  program. 

Although  this  level  of  reporting  can  pro¬ 
vide  quick  review  of  overall  risk  status  for 
identified  problems,  more  detailed  risk 
planning  and  status  can  be  provided  on 
individual  risk  items.  For  example,  some 
program  IPTs  have  combined  risk  level  and 
scheduled  activities  to  provide  a  graphical 


overview  of  risk  status  for  either  internal 
or  external  review.  One  method  for  graphi¬ 
cally  showing  risk  status  for  an  individual 
item  is  shown  in  Figure  5-16. 

5.7.7  Management  Indicator  System 

5.7.7.1  Description.  A  management  indi¬ 
cator  system  is  a  set  of  indicators  or  metrics 
that  provide  the  PMO  with  timely  infor¬ 
mation  on  the  status  of  the  program  and 
risk-handling  actions,  and  is  essential  to 
risk  monitoring  and  program  success.  To 
be  meaningful,  these  metrics  should  have 
some  objective  value  against  which  ob¬ 
served  data  can  be  measured,  reflecting 
trends  in  the  program  or  lack  thereof. 
Metrics  should  be  developed  jointly  by  the 
PMO  and  the  contractor.  The  contractor's 
approach  to  metrics  should  be  a  consider¬ 
ation  in  the  proposal  evaluation  process. 
If  the  contractor  does  not  have  an  estab¬ 
lished  set  of  metrics,  this  may  be  an  area  of 
risk  that  will  need  to  be  addressed. 


Risk  Management  Status 

Risk 
Ran  # 

Risk  Rrofiie 

High 

Moderate  Low 

Status/Comment 

Da^  stm  In  review;  need 
to  assign  part  numbers. 

94-12-9 

Non-stock  Listed  Spares 

( 

{ _ ) 

Data  reviewed;  updates  not 
required  attNstime. 

94-12-10 

Engineering  Updates 

c 

D  C 

- ^Closed  ) 

94-12-11 

Spares  &  Support 

( 

3  C 

- Closed  ) 

Spares  listing  approved  In 

94-12-12 

Long  Lead  Requisitions 

( 

J  C 

' '  *  )  C  ) 

definltlzatlon  conference.  No 

current  abatement  plan. 

94-12-13 

T.O.  Validation 

( 

D  C 

- )-|^  Closed  ) 

Closed  Issue 

Contractor  LSA  plan  submitted 

94-12-14 

Lack  of  LSA  Reconls  for  GFE 

( 

D  € 

C  ) 

for  approval;  rescheduled  for 

5f95. 

Analysis  in  work,  identifying 
last  opportunity  buys. 

Studying  Commercial  Mux 

Interface. 

Questions  about  antenna 
location  and  cable  raised  risk. 

94-12-15 

Program  Parts  Obsolescence 

( 

D  C 

. 

94-12-51 

Design  Maturity 

c 

D  C 

- Closed  ) 

94-12-16 

System  Y  Interface  Definition 

c 

D  i 

c  ) 

Figure  5-15.  Example  Showing  Detailed  List  of  Top-Level  Risk  Information 
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LACK  OF  SUPPORT  RECORDS  FOR  GFE 


Figure  5-16.  Example  of  More  Complex  Combination  of  Risk  Level  and 

Scheduled  Tasks 


and  support,  and  within  these  groups  as 
either  product-  or  process-related.  Prod¬ 
uct-related  metrics  pertain  to  characteristics 
of  the  system  being  developed;  they  can  in¬ 
clude  such  things  as  planned  and  demon¬ 
strated  values  of  the  critical  parameters  moni¬ 
tored  as  part  of  the  TPM  process  and  sys¬ 
tem-unique  data  pertaining  to  the  different 
steps  in  the  development  and  acquisition 
processes.  Table  5-4  provides  examples  of 
product-related  metrics. 

Process  metrics  pertain  to  the  various  pro¬ 
cesses  used  in  the  development  and  produc¬ 
tion  of  the  system.  For  each  program,  cer¬ 
tain  processes  are  critical  to  the  achievement 
of  program  objectives.  Failure  of  these  pro¬ 
cesses  to  achieve  their  requirements  is  sjnnp- 
tomatic  of  significant  problems.  Metrics  data 
can  be  used  to  diagnose  and  aid  in  problem 
resolution.  They  should  be  used  in  formal, 
periodic  performance  assessments  of  the 


various  development  processes  and  to  evalu¬ 
ate  how  well  the  system  development  pro¬ 
cess  is  achieving  its  objectives.  DoD4245.7M, 
Transition  from  Development  to  Production,  and 
other  supporting  documents  such  as  N  AVSO 
P-6071,  Best  Practices,  identify  seven  process 
areas:  funding,  design,  test,  production,  fa¬ 
cilities,  logistics,  and  management.  Within 
each  of  these  areas,  a  number  of  specific  pro¬ 
cesses  are  identified  as  essential  to  assess, 
monitor,  and  establish  program  risk  at  an  ac¬ 
ceptable  level;  the  documents  also  provide 
risk  indicators  that  can  be  used  as  the  basis 
for  selecting  specific  process  metrics.  An¬ 
other  document.  Methods  and  Metrics  for  Prod¬ 
uct  Success,  July  1994,  published  by  the  Of¬ 
fice  of  the  Assistant  Secretary  of  the  Navy 
(RD&A),  Product  Integrity  Directorate,  pro¬ 
vides  a  set  of  metrics  for  use  in  assessing  and 
monitoring  the  design,  test,  and  production 
risk  areas.  Table  5-5  provides  examples  of 
process-related  metrics. 
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Table  5-4.  Examples  of  Product-Related  Metrics 


Engineering 

Requirements 

Production 

Support 

Key  Design 

Parameters 

Weight 

Size 

Endurance 

Range 

Design  Maturity 

Open  problem 
reports 

Number  of 
engineering  change 
proposals 

Number  of  drawings 
released 

Failure  activities 

Computer  Resource 
Utilization 

Requirements 

Traceability 

Requirements  Stability 

Manufacturing  Yields 

Incoming  Material 
Yields 

Delinquent 

Requisitions 

Unit  Production  Cost 

Process  Proofing 

Special  Tools  and 

Test  Equipment 

Support 

Infrastructure 

Footprint 

Manpower  Estimates 

(RD&A),  Product  Integrity  Directorate,  pro¬ 
vides  a  set  of  metrics  for  use  in  assessing  and 
monitoring  the  design,  test,  and  production 
risk  areas.  Table  5-5  provides  examples  of 
process-related  metrics. 

Cost  and  schedule  metrics  can  be  used  to 
depict  how  the  program  is  progressing  to¬ 
ward  completion.  The  information  pro¬ 
vided  by  the  contractor  in  the  earned  value 
management  system  can  serve  as  these 
metrics,  showing  how  the  actual  work  ac¬ 
complished  compares  with  the  work 
planned  in  terms  of  schedule  and  cost. 
Other  sources  of  cost  and  schedule  metrics 
include  the  contractor's  cost  accounting 
information  and  the  integrated  master 
schedule.  Table  5-6  provides  examples  of 
cost  and  schedule  metrics. 

5.8  RISK  MANAGEMENT 

INFORMATION  SYSTEMS  AND 


DOCUMENTATION 
5.8.1  Description 

To  manage  risk,  PMs  should  have  a  data¬ 
base  management  system  that  stores  and 
allows  retrieval  of  risk-related  data.  The 
risk-management  information  system  pro¬ 
vides  data  for  creating  reports  and  serves 
as  the  repository  for  all  current  and  histori¬ 
cal  information  related  to  risk.  This  infor¬ 
mation  may  include  risk  assessment  docu¬ 
ments,  contract  deliverables,  if  appropri¬ 
ate,  and  any  other  risk-related  reports.  The 
PM  should  consider  a  number  of  factors  in 
establishing  the  management  information 
system  and  developing  rules  and  proce¬ 
dures  for  the  reporting  system: 

•  Assign  management  responsibility  for 
the  reporting  system 

•  Publish  any  restrictions  for  entering 
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Table  5-5.  Examples  of  Process  Metrics 


Design 

Requirements 

Trade 

Studies 

Design 

Process 

integrated 
Test  Pian 

Failure 

Reporting 

System 

Manufacturing 

Plan 

Development 
of  require¬ 
ments  trace- 
ability  plan 

Development 
of  specification 
tree 

Specifications 
reviewed  for: 

Definition  of 
all  use  envi¬ 
ronments 

Definition  of 
all  functional 
requirements 
for  each 
mission 
performed 

Users  needs 
prioritized 

Alternative 
system  con¬ 
figurations 
selected 

Test 

methods 

selected 

Design 

requirements 

stability 

Producibility 

analysis 

conducted 

Design 
analyzed  for: 

Cost 

Parts 

reduction 

Manufac¬ 

turability 

Testability 

All  develop¬ 
mental  tests 
at  system  and 
subsystem 
level 
identified 

Identification 
of  who  will  do 
test 

(Government, 

contractor, 

supplier) 

Contractor 

corporate- 

level 

management 
involved  in 
failure 

reporting  and 
corrective 
action 
process 

Responsibility 
for  analysis 
and  corrective 
action  as¬ 
signed  to  spe¬ 
cific  individual 
with  close-out 
date 

Plan 

documents 
methods  by 
which  design  to 
be  built 

Plan  contains 
sequence  and 
schedule  of 
events  at  con¬ 
tractor  and  sub¬ 
contractor 
levels  that 
defines  use  of 
materials,  fabri¬ 
cation  flow,  test 
equipment, 
tools,  facilities, 
and  personnel 

Reflects  manu¬ 
facturing  inclu¬ 
sion  in  design 
process.  In¬ 
cludes  identifi¬ 
cation  and 
assessment  of 
design  facilities 

data  into  the  database 

•  Identify  reports  and  establish  a  sched¬ 
ule,  if  appropriate 


Table  5-6.  Examples  of  Cost  and  Schedule 
Metrics 


Cost 

Schedule 

Cost  variance 

Cost  performance 
index 

Estimate  at 
completion 

Management 

reserve 

Schedule  variance 

Schedule  performance 
index 

Design  Schedule 
Performance 

Manufacturing  Schedule 
Performance 

Test  Schedule 

Performance 

•  Use  standard  report  formats  as  much 
as  possible 

•  Ensure  that  the  standard  report  for¬ 
mats  support  all  users,  such  as  the  PM, 
IPTs,  and  IIPTs 

•  Establish  policy  concerning  access  to 
the  reporting  system  and  protect  the  data¬ 
base  from  unauthorized  access. 

With  a  well-structured  information  system, 
a  PMO  may  create  reports  for  senior  man¬ 
agement  and  retrieve  data  for  day-to-day 
program  management.  Most  likely,  the  PM 
will  choose  a  set  of  standard  reports  that  suits 
specific  needs  on  a  periodic  basis.  This  eases 
definition  of  the  contents  and  structure  of  the 
database.  In  addition  to  standard  reports. 
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Figure  5-17.  Conceptual  Risk  Management  and  Reporting  System 


the  PMO  will  need  to  create  ad  hoc  reports 
in  response  to  special  queries,  etc.  Commer¬ 
cial  database  programs  now  available  allow 
the  PMO  to  create  reports  with  relative  ease. 
Figure  5-17  shows  a  concept  for  a  manage¬ 
ment  and  reporting  system. 

5.8.2  Risk  Management  Reports 

The  following  are  examples  of  basic  reports 
that  a  PMO  may  use  to  manage  its  risk  pro¬ 
gram.  Each  office  should  tailor  and  am¬ 
plify  them,  if  necessary,  to  meet  specific 
needs. 

Risk  Information  Form.  The  PMO  needs 
a  document  that  serves  the  dual  purpose 
of  a  source  of  data  entry  information  and  a 
report  of  basic  information  for  the  IPTs.  The 
RIF  serves  this  purpose.  It  gives  members 
of  the  project  team,  both  Government  and 
contractors,  a  format  for  reporting  risk-re¬ 
lated  information.  The  RIF  should  be  used 
when  a  potential  risk  event  is  identified 
and  updated  over  time  as  information  be¬ 
comes  available  and  the  status  changes.  As 
a  source  of  data  entry,  the  RIF  allows  the 
database  administrator  to  control  entries. 
To  construct  the  database  and  ensure  the 
integrity  of  data,  the  PMO  should  design  a 


standard  format  for  a  RIF. 

Risk  Assessment  Report  Risk  assessments 
form  the  basis  for  many  program  decisions, 
and  the  PM  will  probably  need  a  detailed 
report  of  any  assessment  of  a  risk  event.  A 
Risk  Assessment  Report  (RAR)  is  prepared 
by  the  team  that  assessed  a  risk  event  and 
amplifies  the  information  in  the  RIF.  It  docu¬ 
ments  the  identification  and  analysis  process 
and  results.  The  RAR  provides  information 
for  the  summary  contained  in  the  RIF,  is  the 
basis  for  developing  risk-handling  plans,  and 
serves  as  a  historical  recording  of  program 
risk  assessment.  Since  RARs  may  be  large 
documents,  they  may  be  stored  as  files.  RARs 
should  include  information  that  links  it  to 
the  appropriate  RIF. 

Risk-Handling  Documentation.  Risk¬ 
handling  documentation  may  be  used  to 
provide  the  PM  with  the  information  he 
needs  to  choose  the  preferred  mitigation 
option  and  is  the  basis  for  the  handling  plan 
summary  that  is  contained  in  the  RIF.  This 
document  describes  the  examination  pro¬ 
cess  for  the  risk-handling  options  and  gives 
the  basis  for  the  selection  of  the  recom¬ 
mended  choice.  After  the  PM  chooses  an 
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option,  the  rationale  for  that  choice  may  be 
included.  There  should  be  a  plan  for  each 
risk-mitigation  task.  Risk-handHng  plans 
are  based  on  results  of  the  risk  assessment. 
This  document  should  include  information 
that  hnks  it  to  the  appropriate  RIF. 

Risk  Monitoring  Documentation.  The  PM 
needs  a  summary  document  that  tracks  the 
status  of  high  and  moderate  risks.  He  can 
produce  a  risk-tracking  list,  for  example,  that 
uses  information  that  has  been  entered  from 
the  RBF.  Each  PMO  should  tailor  the  track¬ 
ing  list  to  suit  its  needs.  If  elements  of  needed 
information  are  not  included  in  the  RIF,  they 
should  be  added  to  that  document  to  ensure 
entry  into  the  database. 

Database  Management  System  (DBMS). 
The  DBMS  that  the  PM  chooses  may  be 
commercial.  Government-owned,  or  con¬ 
tractor-developed.  It  should  provide  the 
means  to  enter  and  access  data,  control  ac¬ 
cess,  and  create  reports.  Many  options  are 
available  to  users. 

Key  to  the  MIS  are  the  data  elements  that 
reside  in  the  database.  The  items  hsted  in 
Table  5-7  are  examples  of  risk  information 
that  might  be  included  in  a  database  that 
supports  risk  management.  They  are  a 
compilation  of  several  risk  reporting  forms 
used  in  current  DoD  programs  and  other 
risk  document  sources.  "Element"  is  the 
title  of  the  database  field;  "Description"  is 
a  summary  of  the  field  contents.  PMs 
should  tailor  the  list  to  suit  their  needs. 

5.9  SOFTWARE  RISK 
MANAGEMENT 
METHODOLOGIES 

The  management  of  risk  in  software  inten¬ 
sive  programs  is  essentially  the  same  as  for 
any  other  type  of  program.  A  number  of 
methodologies  specifically  focus  on  the 
software  aspects  of  developmental  pro¬ 
grams  and  can  be  useful  in  identifying  and 


analyzing  risks  associated  with  software. 
Several  of  these  methodologies  are  de¬ 
scribed  in  the  U.S.  Air  Force  publication. 
Guide  to  Software  Acquisition  and  Manage¬ 
ment.  Three  of  these  methodologies  are 
described  below. 

5.9.1  Software  Risk  Evaluation  (SRE) 

This  is  a  formal  approach  developed  by  the 
Software  Engineering  Institute  (SEI)  using 
a  risk  management  paradigm  that  defines 
a  continuous  set  of  activities  to  identify, 
communicate,  and  resolve  software  risks. 
These  activities  are  to  identify,  analyze, 
plan,  track,  and  control.  (The  SEI  activi¬ 
ties  are  analogous  to  the  activities  of  the 
risk  management  process  defined  in  this 
section.) 

This  methodology  is  initiated  by  the  PM, 
who  tasks  an  independent  SRE  team  to 
conduct  a  risk  evaluation  of  the  contractor's 
software  development  effort.  The  team  ex¬ 
ecutes  the  following  SRE  functions  in  per¬ 
forming  this  evaluation,  and  prepares  find¬ 
ings  that  will  provide  the  PM  with  the  re¬ 
sults  of  the  evaluation: 

•  Detection  of  the  software  technical 
risks  present  in  the  program.  An  SEI  Tax¬ 
onomy-Based  Questionnaire  is  used  to  en¬ 
sure  that  all  areas  of  potential  risk  are  iden¬ 
tified.  This  questionnaire  is  based  on  the 
SEI  Software  Development  Risk  Taxonomy, 
which  provides  a  systematic  way  of  orga¬ 
nizing  and  eliciting  risks  within  a  logical 
framework. 

•  Specification  of  all  aspects  of  identi¬ 
fied  technical  software  risks,  including 
their  conditions,  consequences,  and  source. 

•  Assessment  of  the  risks  to  determine 
the  probability  of  risk  occurrence  and  the 
severity  of  its  consequences. 

•  Consolidation  of  the  risk  data  into  a 
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Table  5-7.  Data  Base  Management  System  Elements 


Element 

Description 

Risk  Identification 
(ID)  Number 

Identifies  the  risk  and  is  a  critical  element  of  information,  assuming  that 
a  relational  database  will  be  used  by  the  PMO.  (Construct  the  ID 
number  to  identify  the  organization  responsible  for  oversight.) 

Risk  Event 

States  the  risk  event  and  identifies  it  with  a  descriptive  name.  The 
statement  and  risk  identification  number  will  always  be  associated  in  any 
report. 

Priority 

Reflects  the  importance  of  this  risk  priority  assigned  by  the  PMO 
compared  to  all  other  risks,  e.g.,  a  one  (1)  indicates  the  highest  priority. 

Date  Submitted 

Gives  the  date  that  the  RIF  was  submitted. 

Major  System/ 
Component 

Identifies  the  major  system/component  based  on  the  WBS. 

Subsystem/ 
Functionai  Area 

Identifies  the  pertinent  subsystem  or  component  based  on  the  WBS. 

Category 

Identifies  the  risk  as  technical/performance  cost  or  schedule  or 
combination  of  these. 

Statement  of  Risk 

Gives  a  concise  statement  (one  or  two  sentences)  of  the  risk. 

Description  of 
Risk 

Briefly  describes  the  risk.  Lists  the  key  processes  that  are  involved  in 
the  design,  development,  and  production  of  the  particular  system  or 
subsystem.  If  technical/performance,  includes  how  it  is  manifested 
(e.g.,  design  and  engineering,  manufacturing,  etc. 

Key  parameters 

Identifies  the  key  parameter,  minimum  acceptable  value,  and  goal 
value,  if  appropriate.  Identifies  associated  subsystem  values  required  to 
meet  the  minimum  acceptable  value  and  describes  the  principal  events 
planned  to  demonstrate  that  the  minimum  value  has  been  met. 

Assessment 

States  if  an  assessment  has  been  done.  Cites  the  Risk  Assessment 
Report,  if  appropriate. 

Analyses 

Briefly  describes  the  analysis  done  to  assess  the  risk.  Includes  rationale 
and  basis  for  results. 

Probability  of 
Occurrence 

States  the  likelihood  of  the  event  occurring,  based  on  definitions  in  the 
program’s  risk-management  plan. 

Consequence 

States  the  consequence  of  the  event,  if  it  occurs,  based  on  definitions  in 
the  program’s  risk-management  plan. 

Time  Sensitivity 

Estimates  the  relative  urgency  for  implement  the  risk-handling  option. 

Other  Affected 

Areas 

If  appropriate,  identifies  any  other  subsystem  or  process  that  this  risk 
affects. 

Risk  Handling  Plans 

Briefly  describes  plans  to  mitigate  the  risk.  Refers  to  any  detailed  plans 
that  may  exist,  if  appropriate. 

Risk  Monitoring 
Activity 

Measures  using  metrics  for  tracking  progress  in  implementing  risk¬ 
handling  plans  and  achieving  planned  results  for  risk  reduction. 

Status 

Briefly  reports  the  status  of  the  risk-handling  activities  and  outcomes 
relevant  to  any  risk-handling  milestones. 

Status  Date 

Lists  date  of  the  status  report. 

Assignment 

Lists  individual  assigned  responsibility  for  mitigation  activities. 

Reported  By 

Records  name  and  phone  number  of  individual  who  reported  the  risk. 
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concise  format  suitable  for  decision  making. 

A  detailed  discussion  of  the  SRE  method¬ 
ology  is  found  in  SEI  Technical  Report 
CMU/SEI-94-TR-19,  Software  Risk  Evalua¬ 
tion  Model,  Version  1.0,  December  1994. 

5.9.2  Boehm's  Software  Risk 
Management 

This  risk  management  methodology,  devel¬ 
oped  by  Barry  W.  Boehm  and  described  in 
IEEE  Software,  Software  Risk  Management: 
Principles  and  Practices,  January  1991,  con¬ 
sists  of  two  primary  steps,  each  with  three 
subordinate  steps.  This  risk  management 
structure  is  shown  in  Table  5-8. 

Boehm  provides  a  number  of  techniques 
that  can  be  used  to  accomplish  each  of  the 
steps  in  the  methodology.  For  example,  to 


assist  in  risk  identification,  he  includes  the 
top  10  top-level  software  risks,  based  on 
surveys  of  experienced  software  project 
managers.  These  risks  are  shown  in  Table 
5-9,  along  with  recommended  techniques 
to  manage  them.  Using  this  list  as  a  start¬ 
ing  point,  managers  and  engineers  can  then 
develop  lists  of  lower  level  risks  to  be  as¬ 
sessed  and  resolved. 

5.9.3  Best  Practices  Initiative  Risk 
Management  Method 

The  Software  Acquisition  Best  Practices 
Initiative  was  instituted  in  1994  to  improve 
and  restructure  the  software  acquisition 
management  process  through  the  identifi¬ 
cation  of  effective  practices  used  in  success¬ 
ful  software  developments.  One  result  of 
this  effort  was  the  publication  of  the  Pro¬ 
gram  Manager's  Guide  to  Software  Acquisition 


Table  5-8.  Software  Risk  Management  Steps 


Primary  Steps 

Secondary  Steps 

Description 

Risk  Assessment 

Risk  Identification 

-  Produces  lists  of  project 
specific  risk  events 

Risk  Analysis 

-  Assesses  probability  of  risk 
event  and  consequences 

-  Assesses  compound  risk 
resulting  from  risk  event 
interaction 

Risk  Prioritization 

-  Produces  rank-ordered  list  of 
identified  and  analyzed  risk 
events 

Risk  Control 

Risk  Management  Planning 

-  Produces  plan  for  addressing 
each  risk  event 

-  Integrates  individual  risk 
event  plans  with  each  other 
and  the  overall  plan 

Risk  Resolution 

-  Establishes  the  environment 
and  actions  to  resolve  or 
eliminate  risks 

Risk  Monitoring 

-  Tracks  progress  in  resolving 
risks 

-  Provides  feedback  for 
refining  prioritization  and 
plans 
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Table  5-9.  Top  10  Software  Risks 


Risk 

Risk  Management  Techniques 

Personnel  Shortfalls 

Staffing  with  top  talent;  job  matching;  team 
building;  key  personnel  agreements;  cross 
training 

Unrealistic  schedules  and  budgets 

Detaiied  multisource  cost  and  schedule 
estimation;  design-to-cost;  incremental 
development;  software  reuse;  requirements 
scrubbing 

Developing  the  wrong  software  functions 

Organizational  analysis;  mission  analysis; 
operations  concept  formuiation;  user  surveys; 
prototyping;  early  users’  manuals 

Deveioping  wrong  user  interface 

Task  analysis;  prototyping;  scenarios;  user 
characterization  (functionality,  style,  workload) 

Goldplating 

Requirements  scrubbing;  prototyping; 
cost/benefit  analysis;  design-to-cost 

Continuing  stream  of  requirements  changes 

High  change  threshold;  information  hiding; 
incremental  development  (defer  changes  to 
later  increments) 

Shortfalls  in  externaliy  furnished  components 

Benchmarking;  inspections;  reference 
checking;  compatibility  anaiysis 

Shortfaiis  in  externally  performed  tasks 

Reference  checking;  pre-award  audits;  award- 
fee  contracts;  competitive  design  or 
prototyping;  team  buiiding 

Real-time  performance  shortfalls 

Simulation;  benchmarking;  modeling; 
prototyping;  instrumentation;  tuning 

Straining  computer  science  capabilities 

Technical  analysis;  cost-benefit  analysis; 
prototyping;  reference  checking 

Best  Practices  by  the  Software  Program 
Managers  Network  (SPMN).  This  docu¬ 
ment  identified  nine  principal  best  prac¬ 
tices  that  are  essential  to  the  success  of  any 
large-scale  software  development.  The  first 
of  these  nine  is  formal  risk  management. 
To  assist  in  implementing  this  top  practice, 
SPMN  developed  a  three-part  methodol¬ 
ogy  consisting  of  the  following  steps:  ad¬ 
dress  the  problem;  practice  essentials;  and 


check  status.  Specific  activities  associated 
with  these  steps  are  shown  in  Table  5-10. 

SPMN  provides  PMOs  with  specialized 
training  programs  covering  the  core  disci¬ 
plines  and  techniques  for  implementing  this 
formal  risk  management  practice,  as  well  as 
the  other  best  practices.  SPMN  also  has  avail¬ 
able  (or  imder  development)  a  number  of 
guidebooks  designed  to  provide  software 
developers  and  Program  Managers  with 
practical  guidance  for  plarming,  implement¬ 
ing,  and  monitoring  their  programs. 

SPMN  can  be  contacted  at  (703)  521-5231, 
or  on  the  Internet  at  http://spmn.com/ 
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Table  5-10.  Best  Practices  Initiative  Risk  Management  Method 


Best 

Practices  Initiatives  Risk  Management  Steps 

Address  the  Problem 

Practice  Essentials 

Check  Status 

-  Recognize  that  all 
software  has  risk 

-  Attempt  to  resolve  risk 
as  early  as  possible 
when  cost  impact  is 
less  than  it  will  be  later 
in  development 

-  Identify  risks 

-  Decriminalize  risk 

-  Plan  for  risk 

-  Formally  designate  a  Risk 

Officer 

•  Include  in  budget  and  schedule 
a  risk  reserve  buffer  of  time, 
money,  and  other  resources 

-  Compile  database  for  all  non- 
negligible  risks 

-  Prepare  profile  for  each  risk 
showing  probability  and 
consequences 

-  Include  all  risks  over  full  life 
cycle 

-  Provide  frequent  risk  status 
reports  that  include 

-  Top  10  risk  items 

-  Number  of  risk  items 
resolved 

“  Number  of  new  risk  items 

-  Number  of  risk  items 
unresolved 

-  Unresolved  risk  items  on 
critical  path 

-  Probable  costs  for  unresolved 
risks 

-  Risk  Officer  appointed? 

-  Risk  database  set  up? 

-  Risk  assessments  have  clear 
impact  on  program  plans  and 
decisions? 

-  Frequency  and  timeliness  of 
risk  assessment  updates 
consistent  with  decision 
updates? 

-  Objective  criteria  used  to 
identify,  assess,  and  manage 
risk? 

-  Information  flow  patterns  and 
reward  criteria  support 
identification  of  risk  by  all 
program  personnel? 

-  Risks  identified  throughout 
entire  life  cycle? 

-  Risk  management  reserve 
exist? 

-  Risk  profile  for  every  risk,  and 
components  updated 
regularly? 

-  Risk  management  plan  has 
explicit  provisions  for  alerting 
decision  makers  when  risk 
becomes  imminent? 
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APPENDIX  A 

DoD  RISK  MANAGEMENT  POLICIES  AND  PROCEDURES 


DoD  policies  and  procedures  that  address 
risk  management  for  acquisition  programs 
are  contained  in  four  key  documents: 

•  DoD  Directive  (DoDD)  5000.1,  De¬ 
fense  Acquisition 

•  DoD  Regulation  5000.2-R,  Mandatory 
Procedures  for  Major  Defense  Acquisition 
(MDAPS)  and  Major  Automated  Information 
System  (MAB)  Acquisition  Programs 

•  DoDD  5000.4,  OSD  Cost  Analysis  Im¬ 
provement  Group 

•  DoD  Manual  5000.4-M,  Cost  Analysis 
Guidance  and  Procedures 

The  relevant  sections  of  each  document  are 
referenced  in  the  Defense  Acquisition 
Deskbook  under  Mandatory  Direction  and 
are  displayed  under  DoD-Wide  Practices. 
They  present  some  strong  statements  on 
risk  management  but  collectively  are  not 
sufficient  to  enable  the  establishment  of  an 
effective  risk  management  program.  The 
following  are  verbatim  extracts  of  sections 
of  the  DoD  5000  series  of  documents  that 
address  risk  management  as  part  of  acqui¬ 
sition  policy  and  procedures. 

1.  DoDD  5000.1  Defense  Acquisition, 
March  1996 

Para  D.l.a  Integrated  Management  Frame¬ 
work 

"...  The  acquisition  management  system 
governed  by  this  Directive  provides  for  a 
streamlined  management  process  that  em¬ 
phasizes  risk  management  and  afford¬ 
ability  and  that  explicity  links  milestone 
decisions  to  demonstrated  accomplish¬ 
ments  ... ." 


Para  D.l  .d  Risk  Assessment  and  Manage¬ 
ment 

"PMs  and  other  acquisition  managers  shall 
continually  assess  program  risks.  Risks 
must  be  well  understood,  and  risk  man¬ 
agement  approaches  developed,  before 
decision  authorities  can  authorize  a  pro¬ 
gram  to  proceed  into  the  next  phase  of  the 
acquisition  process.  To  assess  and  manage 
risk,  PMs  and  other  acquisition  managers 
shall  use  a  variety  of  techniques,  includ¬ 
ing  technology  demonstrations, 
prototyping,  and  test  and  evaluation.  Risk 
management  encompasses  identification, 
mitigation,  and  continuous  tracking  and 
control  procedures  that  feed  back  through 
the  program  assessment  process  to  decision 
authorities.  To  ensure  an  equitable  and  sen¬ 
sible  allocation  of  risk  between  government 
and  industry,  PMs  and  other  acquisition 
managers  shall  select  a  contracting  ap¬ 
proach  appropriate  to  the  type  of  system 
being  acquired." 

Para  D.l. a  Event-Oriented  Management 

"The  Department  shall  use  a  rigorous, 
event-oriented  management  process  that 
emphasizes  effective  acquisition  planning, 
improved  and  continuous  communications 
with  users,  and  prudent  risk  management 
by  both  the  government  and  industry. 
Event-oriented  means  that  the  manage¬ 
ment  process  shall  be  based  on  significant 
events  in  the  acquisition  life-cycle  and  not 
arbitrary  calendar  dates." 

Para  D.l.f  Modeling  and  Simulation 

"Models  and  simulations  shall  be  used  to 
reduce  the  time,  resources,  and  risks  of  the 
acquisition  process  and  to  increase  the  qual- 
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ity  of  the  systems  being  acquired.  Represen¬ 
tations  of  proposed  systems  (virtual  proto- 
t)q)es)  shall  be  embedded  in  realistic  syn¬ 
thetic  environments  to  support  the  various 
phases  of  the  acquisition  process,  from  re¬ 
quirements  determination  and  initial  concept 
exploration  to  the  manufacturing  and  test¬ 
ing  of  new  systems,  and  related  training." 

ParaD3.e  Tailoring 

"...MDAs  shall  promote  flexible,  tailored 
approaches  to  oversight  and  review  based 
on  mutual  trust  and  a  program's  size,  risk, 
and  complexity." 

2.  DoD  Regulation  (DoD)  5000.2-R.  Man¬ 
datory  Procedures  for  Major  Defense  Ac¬ 
quisition  Programs  (MDAPs)  and  Major 
Automated  Information  System  (MAIS) 
Acquisition  Programs,  March  15, 1996 

Para  1.1  Purpose 

"...Any  singular  MDAP  or  MAIS  need  not 
follow  the  entire  process  described  below. 
However,  cognizant  of  this  model,  the  Pro¬ 
gram  Manager  (PM)  and  the  Milestone 
Decision  Authority  (MDA)  shall  structure 
the  MDAP  or  MAIS  to  ensure  a  logical  pro¬ 
gression  through  a  series  of  phases  de¬ 
signed  to  reduce  risk,  ensure  affordability, 
and  provide  adequate  information  for  de¬ 
cision  making  that  will  be  provide  the 
needed  capability  to  the  warfighter  in  the 
shortest  practical  time." 

Para  1.2  Overview  of  the  Acquisition 
Management  Process 

"...  The  acquisition  process  shall  be  struc¬ 
tured  in  logical  phases  separated  by  major 
decision  points  called  milestones.  The  pro¬ 
cess  shall  begin  with  the  identification  of 
broadly  stated  mission  needs  that  cannot 
be  satisfied  by  nonmateriel  solutions.  Ac¬ 
quisition  program  stakeholders  shall  con¬ 
sider  the  full  range  of  alternatives  prior  to 
deciding  to  initiate  a  new  MDAP  or  MAIS. 


Threat  projections,  system  performance, 
unit  production  cost  estimates,  life-cycle 
costs,  interoperability,  cost-performance- 
schedule  trade-offs,  acquisition  strategy, 
affordability  constraints,  and  risk  manage¬ 
ment  shall  be  major  considerations  at  each 
milestone  decision  point,  including  the 
decision  to  start  a  new  program." 

"...  At  program  initiation,  and  after  consid¬ 
eration  of  the  views  of  the  Working-Level 
Integrated  Product  Team  (IPT)  and 
Overarching  IPT  members,  the  PM  shall 
propose,  and  the  MDA  shall  consider  for 
approval,  the  appropriate  milestones,  the 
level  of  decision  for  each  milestone,  and 
the  documentation  needed  for  each  mile¬ 
stone.  This  proposal  shall  consider  the  size, 
complexity,  and  risk  of  the  program.  The 
determinations  made  at  program  initiation 
shall  be  reexamined  at  each  milestone  in 
light  of  then-current  program  conditions." 

Para  1.4  Acquisition  Phases  & 
Accomplishments 

"...  Tailoring  shall  give  full  consideration  to 
applicable  statutes.  The  number  of  phases 
and  decision  points  shall  be  tailored  to  meet 
the  specific  needs  of  individual  PMs,  based 
on  objective  assessments  of  a  program's  cat¬ 
egory  status,  risks,  the  adequacy  of  proposed 
risk  management  plans,  and  the  urgency  of 
the  user's  need.  Tailored  acquisition  strate¬ 
gies  may  vary  the  way  in  which  core  activi¬ 
ties  are  to  be  conducted,  the  formality  of  re¬ 
views  and  documentation,  and  the  need  for 
other  supporting  activities.  ACAT 11  and  HI 
program  managers  shall  work  with  their 
decision  authorities  to  tailor  any  documen¬ 
tation  and  decision  points  to  the  needs  of  the 
individual  program." 

Para  1.4.2  Phase  0:  Concept  Exploration 

"...  Phase  0  typically  consists  of  competitive, 
parallel  short-term  concept  studies.  The  fo¬ 
cus  of  these  efforts  is  to  define  and  evaluate 
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the  feasibility  of  alternative  concepts  and  to 
provide  a  basis  for  assessing  the  relative 
merits  (i.e.,  advantages  and  disadvantages, 
degree  of  risk)  of  these  concepts  at  the  next 
milestone  decision  point..." 

Para  1.43  Phase  I:  Program  Definition 
and  Risk  Reduction 

"During  this  phase,  the  program  shall  be¬ 
come  defined  as  one  or  more  concepts,  de¬ 
sign  approaches,  and/ or  parallel  technolo¬ 
gies  are  pursued  as  warranted.  Assess¬ 
ments  of  the  advantages  and  disadvantages 
of  alternative  concepts  shall  be  refined. 
Prototyping,  demonstrations,  and  early  op¬ 
erational  assessments  shall  be  considered 
and  included  as  necessary  to  reduce  risk 
so  that  technology,  manufacturing,  and 
support  risks  are  well  in  hand  before  the 
next  decision  point.  Cost  drivers,  life-cycle 
cost  estimates,  cost-performance  trades, 
interoperability,  and  acquisition  strategy 
alternatives  shall  be  considered  to  include 
evolutionary  and  incremental  software  de¬ 
velopment." 

Para  3.2.2.23  Cost 

"...  As  the  program  progresses  through 
later  acquisition  phases,  procurement  costs 
shall  be  refined  based  on  contractor  actual 
(or  return)  costs  from  program  definition 
and  risk  reduction,  engineering  and  manu¬ 
facturing  development,  or  from  initial  pro¬ 
duction  lots.  In  all  cases,  the  cost  param¬ 
eters  shall  reflect  the  total  program  and  be 
realistic  cost  estimates,  based  on  a  careful 
assessment  of  risks  and  realistic  apprais¬ 
als  of  the  level  of  costs  most  likely  to  be 
realized.  The  amount  budgeted  shall  not 
exceed  the  total  cost  threshold  estimated 
in  the  APB." 

Para  3.2.3  Exit  Criteria 

"...  Exit  criteria  will  normally  be  selected 
to  track  progress  in  important  technical, 
schedule  or  management  risk  areas." 


Para  3.3  Acquisition  Strategy 

"...  Essential  elements  in  this  context  in¬ 
clude,  but  are  not  limited  to,  sources,  risk 
management,  cost  as  an  independent  vari¬ 
able,  contract  approach,  management  ap¬ 
proach,  environmental  considerations,  and 
source  of  support.  The  PM  shall  also  ad¬ 
dress  other  major  initiatives  that  are  criti¬ 
cal  to  the  success  of  the  program. 

...  The  acquisition  strategy  shall  be  tailored 
to  meet  the  specific  needs  of  individual 
programs,  including  consideration  of  in¬ 
cremental  (block)  development  and  field¬ 
ing  strategies.  The  benefits  and  risk  asso¬ 
ciated  with  reducing  lead  time  through 
concurrency  shall  be  specifically  addressed 
in  tailoring  the  acquisition  strategy." 

Para  3. 3. 1.3  Industrial  Capability 

"The  PM  shall  structure  the  acquisition 
strategy  to  promote  sufficient  program  sta¬ 
bility  to  encourage  industry  to  invest,  plan 
and  bear  risks.  Program  needs  shall  be  met 
through  reliance  on  a  national  technology 
and  industrial  base  sustained  primarily  by 
commercial  demands,  and  minimize  the 
need  for  new  defense-unique  industrial 
capabilities.  Foreign  sources  and  interna¬ 
tional  cooperative  developments  shall  be 
used  where  advantageous  and  within  limi¬ 
tations  of  the  law  (DEARS  Part  225). 

The  program  acquisition  strategy  shall  ana¬ 
lyze  the  industrial  capability  to  design, 
develop,  produce,  support  and,  if  appro¬ 
priate,  restart  the  program.  (10  use  2440). 
This  analysis  shall  identify  DoD  invest¬ 
ments  needed  to  create  new  industrial  ca¬ 
pabilities,  and  the  risks  of  industry  being 
unable  to  provide  program  manufacturing 
capabilities  at  planned  cost  and  schedule." 
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Para  3.3.2  Cost,  Schedule,  and 
Perfonnance  Risk  Management 

"The  PM  shall  establish  a  risk  management 
program  for  each  acquisition  program  to 
identify  and  control  performance,  cost,  and 
schedule  risks.  The  risk  management  pro¬ 
gram  shall  identify  and  track  risk  drivers, 
define  risk  abatement  plans,  and  provide 
for  continuous  risk  assessment  throughout 
each  acquisition  phase  to  determine  how 
risks  have  changed.  Risk  reduction  mea¬ 
sures  shall  be  included  in  cost-performance 
trade-offs,  where  applicable.  The  risk  man¬ 
agement  program  shall  plan  for  back-ups 
in  risk  areas  and  identify  design  require¬ 
ments  where  performance  increase  is  small 
relative  to  cost,  schedule,  and  performance 
risk.  The  acquisition  strategy  shall  include 
identification  of  the  risk  areas  of  the  pro¬ 
gram  and  a  discussion  of  how  the  PM  in¬ 
tends  to  manage  those  risks." 

Para  3.3.3.2  Cost  Management  Incentives 

"RFPs  shall  be  structured  to  incentivize  the 
contractor  to  meet  or  exceed  cost  objectives. 
Whenever  applicable,  risk  reduction 
through  use  of  mature  processes  shall  be  a 
significant  factor  in  source  selection.  For 
industry,  competition  to  win  business, 
along  with  attendant  business  profit,  is  by 
far  the  most  powerful  incentive.  Therefore, 
competition  shall  be  maintained  for  as  long 
as  practicable  in  all  acquisition  programs." 

Para  3.3.4  Contract  Approach 

'The  acquisition  strategy  shall  discuss  the 
types  of  contracts  contemplated  for  each 
succeeding  phase,  including  consider¬ 
ations  of  risk  assessment,  reasonable  risk¬ 
sharing  by  Government  and  contractor(s), 
and  the  incentive  structure  for  contractors 
to  decrease  cost." 


Para  3.3.41  Competition 

"Component  breakout  shall  be  considered 
on  every  program  and  shall  be  done  when 
there  are  significant  cost  savings  (inclusive 
of  Government  administrative  costs),  when 
the  technical  or  schedule  risk  of  furnish¬ 
ing  government  items  to  the  prime  contrac¬ 
tor  is  manageable,  and  when  there  are  no 
other  overriding  Governmental  interests 
(e.g.,  industrial  capability  considerations)." 

Para  3. 3. 5. 6  Information  Sharing  and 
DoD  Oversight 

"DoD  oversight  activities  (i.e.,  contract 
administration  offices,  contracting  offices, 
technical  activities,  and  program  manage¬ 
ment  offices)  shall  consider  all  relevant  and 
credible  information  that  might  mitigate 
risks  and  the  need  for  DoD  oversight  be¬ 
fore  designing  and  applying  direct  DoD 
oversight  of  contractor  operations." 

Para  3.4  Test  and  Evaluation 

'Test  and  evaluation  programs  shall  be 
structured  to  integrate  all  developmental 
test  and  evaluation  (DT&E),  operational 
test  and  evaluation  (OT&E),  live-fire  test 
and  evaluation  (LFT&E),  and  modeling 
and  simulation  activities  conducted  by  dif¬ 
ferent  agencies  as  an  efficient  continuum. 
All  such  activities  shall  be  part  of  a  strat¬ 
egy  to  provide  information  regarding  risk 
and  risk  mitigation,  to  provide  empirical 
data  to  validate  models  and  simulations, 
to  permit  an  assessment,  the  attainment  of 
technical  performance  specifications  and 
system  maturity,  and  to  determine  whether 
systems  are  operationally  effective,  suit¬ 
able,  and  survivable  for  intended  use." 
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Para  3.4.1  Test  and  Evaluation  Strategy 

6.  "Early  testing  of  prototypes  in  Phase  I, 
Program  Definition  and  j^sk  Reduction, 
and  early  operational  assessments  shall  be 
emphasized  to  assist  in  identifying  risks." 

Pflrfl  3.4.2  Developmental 
Test  and  Evaluation 

"Developmental  test  and  evaluation  pro¬ 
grams  shall: ... 

3.  Support  the  identification  and  descrip¬ 
tion  of  design  technical  risks;... 

4.  Assess  progress  toward  meeting  Criti¬ 
cal  Operational  Issues,  mitigation  of  acqui¬ 
sition  technical  risk,  achievement  of  manu¬ 
facturing  process  requirements  and  system 
mahirity; ..." 

Para  3.4.3  Certification  of  Readiness  for 
Operational  Test  and  Evaluation 

"...  Risk  management  metrics,  measures, 
indicators,  and  associated  thresholds  shall 
include  cost,  schedule,  requirements  trace- 
ability  and  fault  profile." 

Para  3.5.1  Life-Cycle  Cost  Estimates 
"The  life-cycle  cost  estimates  shall  be: ... 

4.  Neither  optimistic  nor  pessimistic,  but 
based  on  a  careful  assessment  of  risks  and 
reflecting  a  realistic  appraisal  of  the  level 
of  cost  most  likely  to  be  realized." 

Para  4.3  Systems  Engineering 

"...  The  system  engineering  process  shall 
establish  a  proper  balance  between  perfor¬ 
mance,  risk,  cost,  and  schedule,  employ¬ 
ing  a  top-down  iterative  process  of  require¬ 
ments  analysis,  functional  analysis  and  al¬ 
location,  design  synthesis  and  verification, 
and  system  analysis  and  control." 


"System  Analysis  and  Control.  System 
analysis  and  control  activities  shall  be  es¬ 
tablished  to  serve  as  a  basis  for  evaluating 
and  selecting  alternatives,  measuring 
progress,  and  documenting  design  deci¬ 
sions.  This  shall  include: 

...  The  establishment  of  a  risk  management 
process  to  be  applied  throughout  the  de¬ 
sign  process.  The  risk  management  effort 
shall  address  the  identification  and  evalu¬ 
ation  of  potential  sources  of  technical  risks 
based  on  the  technology  being  used  and 
its  related  design,  manufacturing,  test  and 
support  processes,  risk  mitigation  efforts, 
and  risk  assessment  and  analysis.  Tech¬ 
nology  transition  planning  and  criteria 
shall  be  established  as  part  of  the  overall 
risk  management  effort." 

Para  4.3.5  Software  Engineering 

"Software  shall  be  managed  and  engi¬ 
neered  using  best  processes  and  practices 
that  are  known  to  reduce  cost,  schedule, 
and  technical  risks." 

Para  5.6  Cost  Analysis  Improvement 
Group  (CAIG)  Procedures 

"The  OSD  CAIG  is  established  in  accor¬ 
dance  with  DoDD  5000.4.  The  DoD  Com¬ 
ponent  responsible  for  acquisition  of  a  sys¬ 
tem  shall  work  with  the  CAIG  providing 
cost,  programmatic,  and  technical  informa¬ 
tion  required  to  estimate  costs  and  appraise 
cost  risks,  and  shall  facilitate  visits  of  the 
CAIG  staff  to  the  program  office,  product 
centers,  test  centers,  and  system 
contractor(s) ..." 

Para  6.2.3  Major  Automated  Informa¬ 
tion  System  Quarterly  Report 
DD-C3I(Q)  1799 

"The  quarterly  Major  Automated  Informa¬ 
tion  System  (MAIS)  status  reporting  sys¬ 
tem  is  designed  to  provide  executive  man- 
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agement  at  the  Component  and  OSD  lev¬ 
els  with  the  program  status,  progress,  is¬ 
sues,  risks,  and  risk  reducers.  The  quar¬ 
terly  report  is  essential  to  the  early  identi¬ 
fication  of  problems  and  associated  plans 
to  initiate  corrective  actions.  The  PM  shall 
provide  the  report  to  the  MDA  in  a  timely 
maimer  to  permit  prompt  action  to  address 
reported  issues  and  problems  ..." 

Para  6.4  Contract  Management  Reports* 

"The  reports  prescribed  by  this  section  shall 
be  used  for  all  applicable  defense  contracts 
and  are  required  for  effective  management 
of  defense  acquisitions.  Use  of  electronic 
media  shall  be  required.  The  Work  Break¬ 
down  Structure  (WBS)  used  in  preparing 
the  reports  covered  by  this  section  shall  be 
in  conformance  with  the  program  WBS  (see 
4.4.1).  Except  for  high-cost  or  high-risk 
elements,  the  normal  level  of  reporting 
detail  required  shall  be  limited  to  level 
three  of  the  contract  WBS. 

*  "Not  normally  applicable  to  ACAT  lA 
programs  due  to  the  lower  dollar  value  of 
ACAT  lA  contracts." 

Para  6.4.1  Contractor  Cost  Data 
Reporting  (CCDR) 

"...  The  following  general  policies  guide  the 
preparation  and  submission  of  CCDR  data: 
1.  Level  of  Cost  Reporting.  Routine  re¬ 
porting  will  be  at  the  contract  WBS  level 
three  for  prime  contractors  and  key  sub¬ 
contractors.  In  addition,  detailed  (i.e.,  sub 
level  three)  reporting  will  be  required  only 
for  those  lower  elements  that  address  high 
risk,  high  value,  or  high  technological  in¬ 
terest  areas  of  a  program.  Identifying  these 
additional  elements  is  a  critical  early  as¬ 
signment  for  program  Cost  Program-level 
IPT  (which  may  include  contractor  mem¬ 
bership,  where  appropriate  and  in  accor¬ 
dance  with  applicable  statutes  (see  3.3.1)). 
Each  element  must  be  justified  in  terms  of 


its  contribution  to  efficient  decision-mak- 
mg. 

3.  DoD  Directive  (DoDD)  5000.4,  OSD 
Cost  Analysis  Improvement  Group 
(CAIG),  November  24,1992 

ParaD.l.h  Risk  Assessment 

"The  CAIG  Chair  report,  in  support  of  a 
milestone  review,  shall  include  quantita¬ 
tive  assessments  of  the  risk  in  the  estimate 
of  life-cycle  costs.  In  developing  an  assess¬ 
ment  of  cost  risk,  the  CAIG  shall  consider 
the  validity  of  such  programmatic  assump¬ 
tions  of  the  CARDS  as  EMD  schedules, 
rates  of  utilization  of  test  assets,  produc¬ 
tion  ramp  rates,  and  buy  rates,  consistent 
with  historical  information.  The  CAIG  shall 
also  consider  imcertainties  in  inputs  to  any 
cost  estimating  relationships  used  in  its 
estimates,  as  well  as  the  uncertainties  in¬ 
herent  in  the  calibration  of  the  CERs,  and 
shall  consider  uncertainties  in  the  factors 
used  in  making  any  estimates  by  analogy. 
The  CAIG  shall  consider  cost  and  sched¬ 
ule  risk  implications  of  available  assess¬ 
ments  of  the  program's  technical  risks,  and 
may  include  the  results  in  its  cost-risk  as¬ 
sessments.  The  CAIG  may  consider  infor¬ 
mation  on  risk  provided  by  any  source, 
although  primary  reliance  will  be  on  the 
technical  risk  assessments  that  are  the  re¬ 
sponsibility  of  the  sponsoring  DoD  com¬ 
ponents,  and  of  other  OSD  offices,  in  ac¬ 
cordance  with  their  functional  respon- 
siiblities." 

4.  DoD  5000.4-M.  Cost  Analysis  Guid¬ 
ance  and  Procedures,  December  1992 

Chapter  1:  (Outline  of  CARD 
Basic  Structure) 

Para  1.2.1.x  (..x..)  Subsystem  Description 

"This  series  of  paragraphs  (repeated  for 
each  subsystem)  describes  the  major  equip)- 
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ment  (hard ware /software)  WBS  compo¬ 
nents  of  the  system.  The  discussion  should 
identify  which  items  are  off-the-shelf.  The 
technical  and  risk  issues  associated  with 
development  and  production  of  individual 
subsystems  also  must  be  addressed." 

Chapter  2: 

Para  2.0  Risk 

"This  section  identifies  the  program 
manager's  assessment  of  the  program  and 
the  measures  being  taken  or  planned  to 
reduce  those  risks.  Relevant  sources  of  risk 
include:  design  concept,  technology  devel¬ 
opment,  test  requirements,  schedule,  acqui¬ 
sition  strategy,  funding  availability,  con¬ 
tract  stability,  or  any  other  aspect  that  might 
cause  a  significant  deviation  from  the 
planned  program.  Any  related  external 
technology  programs  (planned  or  on-go¬ 
ing)  should  be  identified,  their  potential 
contribution  to  the  program  described,  and 
their  funding  prospects  and  potential  for 
success  assessed.  This  section  should  iden¬ 
tify  these  risks  for  each  acquisition  phase 
(DEM/VAL,  EMD,  productions  and  de¬ 
ployment,  and  O&S)." 


Para  2.B.9  Sensitivity  Analysis 

"The  sensitivity  of  projected  costs  to  criti¬ 
cal  program  assumptions  shall  be  exam¬ 
ined.  Aspects  of  the  program  to  be  sub¬ 
jected  to  sensitivity  analysis  shall  be  iden¬ 
tified  in  the  DoD  CCA  of  program  assump¬ 
tions.  The  analysis  shall  include  factors 
such  as  learning  curve  assumptions;  tech¬ 
nical  risk,  i.e.,  the  risk  of  more  development 
and/or  production  effort,  changes  in  per¬ 
formance  characteristics,  schedule  alter¬ 
ations,  and  variations  in  testing  require¬ 
ments;  and  acquisition  strategy  (multiyear 
procurement,  dual  sourcing,  etc.)." 

Para  2.C.3  PM  Presentation 

"The  Program  Manager's  designated  repre¬ 
sentative  shall  present  the  CAIG  with  the 
POE  for  each  alternative  under  constniction 
and  explain  how  each  is  derived.  This  pre¬ 
sentation  shall  cover  the  estimates  and  esti¬ 
mating  procedures  at  the  major  subcompo¬ 
nent  level  (e.g.,  airframe,  engine,  major  avi¬ 
onics  subsystem,  etc.).  The  presentation 
should  focus  on  the  items  that  are  cost  driv¬ 
ers  and/or  elements  of  high  cost  risk." 
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APPENDIX  B 

GENERIC  RISK  MANAGEMENT  PLAN 

SAMPLE  RISK  MANAGEMENT  PLAN 
Preface 

DoDD  5000.1  requires  that  "PMs  and  other  acquisition  managers  shall  continually  assess 
program  risks"  and  that  they  "shall  develop  a  contracting  approach  appropriate  to  the 
type  of  system  being  acquired."  Further,  DoD  5000.2-R  states  that  'The  PM  shall  estab¬ 
lish  a  risk  management  program. .  .to  identify  and  control  performance,  cost,  and  schedule 
risks."  Although  the  need  for  a  risk  management  program  and  a  risk  management  pro¬ 
cess  are  addressed  throughout  this  regulation,  there  is  no  requirement  for  a  formal  risk 
management  plan  (RMP).  However,  Program  Managers  (PMs)  have  found  such  a  plan 
necessary  to  focus  properly  on  the  assessment  and  handling  of  program  risk,  a  core  acqui¬ 
sition  management  issue  that  Milestone  Decision  Authorities  (MDAs)  "must  rigorously 
address  at  appropriate  milestones  before  making  program  decisions." 

Attached  is  a  sample  format  and  RMP  that  is  a  compilation  of  several  good  risk  plans  and 
the  results  of  the  DoD  Risk  Management  Working  Group  Study.  It  represents  the  types  of 
information  and  considerations  that  a  plan,  tailored  to  a  specific  program,  might  contain. 
The  sample  is,  admittedly,  more  useful  for  an  ACAT I  or  II  program;  however,  ACAT  III 
and  IV  programs  may  also  use  it  as  a  guide  to  write  a  tailored  plan  to  meet  their  program 
needs.  A  sample  plan  for  these  programs  is  being  written  and  will  be  available  in  the 
future.  The  DoD  Acquisition  Deskbook,  Section  2.5.2,  has  general  guidance  and  advice  in 
all  areas  of  risk  management.  Section  2.5.2.4  of  the  Deskbook  contains  information  con¬ 
cerning  the  development  of  a  risk  management  plan. 

There  is  a  danger  in  providing  a  sample  document.  First  of  all,  because  it  is  written  as  a 
guide  for  a  general  audience,  it  does  not  satisfy  all  of  the  needs  of  any  particular  program. 
Second,  there  is  the  possibility  that  some  prospective  user  will  simply  adopt  the  plan  as 
written,  despite  the  fact  that  it  does  not  fit  his  or  her  program.  We  discourage  this. 

The  reason  for  providing  this  example  is  to  give  PMs  and  their  staffs  a  starting  point  for 
their  own  planning  process.  It  should  stimulate  thought  about  what  has  to  be  done  and 
give  some  ideas  on  how  to  begin  writing  a  plan.  The  sample  plan  contains  more  information 
than  most  program  offices  shotdd  need.  Few  PMs  have  the  resources  for  a  dedicated  risk 
management  effort  as  depicted  in  the  plan.  The  key  to  using  the  sample  plan  is  to  keep 
things  simple  and  tailor  the  plan  to  suit  your  needs,  focusing  on  the  management  of  risk  in  the 
key  critical  areas  of  your  program. 
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The  italicized  text  reflects  the  outline  of  a  risk  management  plan  found  in  the  DoD  Acquisition 
Deskbook  section  2.5.2A,  Figure  2.5.2A-2. 

SAMPLE  FORMAT  FOR  RISK  MANAGEMENT  PLAN 

INTRODUCTION.  This  section  should  address  the  purpose  and  objective  of  the  plan,  and  pro¬ 
vide  a  brief  summary  of  the  program,  to  include  the  approach  being  used  to  manage  the  program, 
and  the  acquisition  strategy. 

PROGRAM  SUMMARY.  This  section  contains  a  brief  description  of  the  program,  including  the 
acquisition  strategy  and  the  program  management  approach.  The  acquisition  strategy  should  ad¬ 
dress  its  linkage  to  the  risk  management  strategy. 

DEFINITIONS.  Definitions  used  by  the  program  office  should  be  consistent  with  DoD  defini¬ 
tions  for  ease  of  understanding  and  consistency.  However,  the  DoD  definitions  allow  program 
managers  flexibility  in  constructing  their  risk  management  programs.  Therefore,  each  program's 
risk  management  plan  may  include  definitions  that  expand  the  DoD  definitions  to  fit  its  particular 
needs.  For  example,  each  plan  should  include,  among  other  things,  definitions  for  the  ratings  used 
for  technical,  schedule  and  cost  risk.  (Discussion  of  risk  rating  is  contained  in  Acquisition  Deskbook 
Section  2. 5. 2.1.) 

RISK  MANAGEMENT  STRATEGY  AND  APPROACH.  Provide  an  overview  of  the  risk  man¬ 
agement  approach,  to  include  the  status  of  the  risk  management  effort  to  date,  and  a  description  of 
the  program  risk  management  strategy.  See  Acquisition  Deskbook  Sections  2.5.2.1  and  2.5.2.3. 

ORGANIZATION.  Describe  the  risk  management  organization  of  the  program  office  and  list  the 
responsibilities  of  each  of  the  risk  management  participants.  See  Acquisition  Deskbook  Section 
2.5.2.3. 

RISK  MANAGEMENT  PROCESS  AND  PROCEDURES.  Describe  the  program  risk  manage¬ 
ment  process  to  be  employed;  i.e.,  risk  planning,  assessment,  handling,  monitoring  and  documen¬ 
tation,  and  a  basic  explanation  of  these  components.  See  Acquisition  Deskbook  Section  2521 .  Also 
provide  application  guidance  for  each  of  the  risk  management  functions  in  the  process.  If  possible, 
the  guidance  should  be  as  general  as  possible  to  allow  the  program's  risk  management  organization 
(e.g.,  IPTs)  flexibility  in  managing  the  program  risk,  yet  specific  enough  to  ensure  a  common  and 
coordinated  approach  to  risk  management.  It  should  address  how  the  information  associated  with 
each  element  of  the  risk  management  process  will  be  documented  and  made  available  to  all  partici¬ 
pants  in  the  process,  and  how  risks  will  be  tracked,  to  include  the  identification  of  specific  metrics  if 
possible. 

RISK  PLANNING.  This  section  describes  the  risk  planning  process  and  provides  guidance  on 
how  it  will  be  accomplished,  and  the  relationship  between  continuous  risk  planning  and  this  RMP. 
Guidance  on  updates  of  the  RMP  and  the  approval  process  to  be  followed  should  also  be  included. 
See  Section  2.5.2.1  of  the  Deskbook  for  information  on  risk  planning. 

RISK  ASSESSMENT.  This  section  of  the  plan  describes  the  assessment  process  and  procedures  for 
examining  the  critical  risk  areas  and  processes  to  identify  and  document  the  associated  risl(^.  It  also 
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summarizes  the  analyses  process  for  each  of  the  risk  areas  leading  to  the  determination  of  a  risk 
rating.  This  rating  is  a  reflection  of  the  potential  impact  of  the  risk  in  terms  of  its  variance  from 
known  Best  Practices  or  probability  of  occurrence,  its  consequence,  and  its  relationship  to  other  risk 
areas  or  processes.  This  section  may  include: 

•  Overview  and  scope  of  the  assessment  process 

•  Sources  of  information 

•  Information  to  be  reported  and  formats 

•  EJescription  of  how  risk  information  is  retained 

•  Assessment  techniques  and  tools  (see  Section  2.5.2.4  of  the  Deskbook) 

RISK  HANDLING.  This  section  describes  the  procedures  that  can  be  used  to  determine  and 
evaluate  various  risk  handling  options,  and  identifies  tools  that  can  assist  in  implementing  the  risk 
handling  process.  It  also  provides  guidance  on  the  use  of  the  various  handling  options  for  specific 
risks. 

RISK  MONITORING.  This  section  describes  the  process  and  procedures  that  will  be  followed  to 
monitor  the  status  of  the  various  risk  events  identified.  It  should  provide  criteria  for  the  selection  of 
risks  to  be  reported  on,  and  the  frequency  of  reporting.  Guidance  on  the  selection  of  metrics  should 
also  be  included. 

RISK  MANAGEMENT  INFORMATION  SYSTEM,  DOCUMENTATION  AND  REPORTS. 

This  section  describes  the  MIS  structure,  rules,  and  procedures  that  will  be  used  to  document  the 
results  of  the  risk  management  process.  It  also  identifies  the  risk  management  documentation  and 
reports  that  will  be  prepared;  specifies  the  format  and  frequency  of  the  reports;  and  assigns  respon¬ 
sibility  for  their  preparation . 

SAMPLE  RISK  MANAGEMENT  PLAN  FOR  THE  XYZ  PROGRAM 


1.0  INTRODUCTION 


1.1  PURPOSE 

This  Risk  Management  Plan  (RMP)  presents  the  process  for  implementing  proactive  risk 
management  as  part  of  the  overall  management  of  the  XYZ  program.  Risk  management 
is  a  program  management  tool  to  assess  and  mitigate  events  that  might  adversely  impact 
the  program  thereby  increasing  the  likelihood  of  success.  This  RMP  will: 

•  Serve  as  a  basis  for  identifying  alternatives  to  achieve  cost,  schedule,  and  perfor¬ 
mance  goals, 

•  Assist  in  making  decisions  on  budget  and  funding  priorities, 

•  Provide  risk  information  for  Milestone  decisions,  and 

•  Allow  monitoring  the  health  of  the  program  as  it  proceeds. 
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The  RMP  describes  methods  for  identifying,  analyzing,  prioritizing,  and  tracking  risk 
drivers;  developing  risk-handling  plans;  and  planning  for  adequate  resources  to  handle 
risk.  It  assigns  specific  responsibilities  for  the  management  of  risk  and  prescribes  the 
documenting,  monitoring,  and  reporting  processes  to  be  followed. 

This  is  the  second  edition  of  the  Risk  Management  Plan  for  the  XYZ  program.  The  initial 
plan  concentrated  on  tasks  leading  to  Milestone  I  (Phase  0);  this  plan  concentrates  on  the 
tasks  leading  to  Milestone  II  (Phase  I).  Subsequent  updates  to  this  RMP  will  shift  focus  to 
the  later  acquisition  phases.  There  are  changes  in  every  area  of  the  plan;  they  include 
refinement  of  the  risk  identification  process.  The  PMO  Risk  Management  Coordinator 
has  been  identified  and  training  of  IPT  members  has  commenced. 

1.2  PROGRAM  SUMMARY 

The  XYZ  program  was  initiated  in  response  to  Mission  Need  Statement  (MNS)  XXX,  dated 
DD-MM-YYYY  and  Operational  Requirements  Document  (ORD),  dated  DD-MM-YYYY. 
It  is  required  to  support  the  fundamental  objective  of  U.S.  defense  policy  as  stated  in 
Defense  Planning  Guidance  (DPG)  and  the  National  Military  Strate^.  The  XYZ  system  is 
based  on  the  need  for  an  integrated  combat  system  to  link  battlefield  decision  makers. 
The  XYZ  mission  areas  are:  (Etelineate  applicable  areas). 

The  XYZ  program  will  develop  and  procure  120  advanced  platforms  to  replace  the  aging 
ABC  platforms  currently  in  the  inventory.  In  order  to  meet  force  structure  objectives,  the 
XYZ  system  must  reach  Initial  Operational  Capability  (IOC)  (four  platforms)  by  FY-07. 
The  program  is  commencing  an  eight-year  EMD  phase  that  will  be  followed  by  a  five- 
year  procurement  phase.  The  objectives  of  the  EMD  phase  are  to  (discuss  the  specific 
objectives  of  this  phase).  The  program  has  Congiessional  interest  and  is  restricted  to  a 
Research  and  Development  funding  ceiling  of  $300  million. 

1.2.1  System  Description 

The  XYZ  will  be  an  affordable,  yet  capable,  platform  taking  advantage  of  technological 
simplification  and  advancements.  The  XYZ  integrated  Combat  System  includes  all  non¬ 
propulsion  electronics  and  weapons.  Subsystems  provide  capabilities  in  combat  control, 
electronic  warfare  support  measures  (ESM),  defensive  warfare,  navigation,  radar,  interior 
communications,  monitoring,  data  transfer,  tactical  support  device,  exterior  communica¬ 
tions,  and  Identification  Friend  or  Foe  (IFF).  Weapons  systems  are  to  be  provided  by  the 
program  offices  that  are  responsible  for  their  development.  The  Mechanical,  and  Electri¬ 
cal  (M&E)  system  comprises.  ...  The  Combat  System,  M&E  systems,  and  subsystems 
provide  the  XYZ  system  with  the  capability  and  connectivity  to  accomplish  the  broad 
range  of  missions  defined  in  the  MNS  and  ORD. 

1.2.2  Acquisition  Strategy 

The  XYZ  program  initial  strategy  is  to  contract  with  one  prime  contractor  in  Program 
Definition/Risk  Reduction  (PDRR)  for  development  of  two  prototype  systems  for  test 
and  design  validation.  Due  to  the  technical  complexity  of  achieving  the  performance 
levels  of  the  power  generation  systems,  the  prime  will  use  two  sub-contractors  for  the 
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engine  development  and  down  select  to  one  producer  prior  to  low  rate  initial  production, 
which  is  scheduled  for  FY-04.  Various  organizations,  such  as  the  Government  Research 
Laboratory  will  be  funded  to  provide  experts  for  assessment  of  specific  areas  of  risk.  The 
program  has  exit  criteria,  included  in  the  list  of  Critical  Program  Attributes  in  Armex  A, 
that  must  be  met  before  progressing  to  the  next  phase. 

1.2.3  Program  Management  Approach 

The  XYZ  program  is  managed  using  the  IPPD  concept,  with  program  integrated  product 
teams  (PIPTs)  established  largely  along  the  hierarchy  of  the  product  work  breakdown 
structure  (WBS).  There  are  also  cost-performance  and  test  Working  IPTs  (WIPTs)  estab¬ 
lished  for  vertical  coordination  up  the  chain  of  command.  The  PM  chairs  a  program 
integrating  IPT  (IIPT)  that  addresses  issues  that  are  not  resolved  at  the  WIPT  level. 


1.3  DEFINITIONS 


1.3.1  Risk 

Risk  is  a  measure  of  the  inability  to  achieve  overall  program  objectives  within  defined 
cost,  schedule,  and  technical  constraints  and  has  two  components:  (1)  the  probability  of 
failing  to  achieve  a  particular  outcome  and  (2)  the  consequences  of  failing  to  achieve  that 
outcome.  For  processes,  risk  is  a  measure  of  the  difference  between  actual  performance  of 
a  process  and  the  known  best  practice  for  performing  that  process. 

1.3.2  Risk  Event 

Risk  e  and  those  events  within  the  XYZ  program  that,  if  they  go  wrong,  could  result  in 
problems  in  the  development,  production,  and  fielding  of  the  system.  Risk  events  should 
be  defined  to  a  level  such  that  the  risk  and  causes  are  understandable  and  can  be  accu¬ 
rately  assessed  in  terms  of  likelihood /probability  and  consequence  to  establish  the  level 
of  risk.  For  processes,  risk  events  are  assessed  in  terms  of  process  variance  from  known 
best  practices  and  potential  consequences  of  the  variance. 

1.3.3  Technical  Risk 

This  is  the  risk  associated  with  the  evolution  of  the  design  and  the  production  of  the  XYZ 
system  affecting  the  level  of  performance  necessary  to  meet  the  operational  requirements. 
TTie  contractor's  and  subcontractors'  design,  test,  and  production  processes  (process  risk) 
influence  the  technical  risk  and  the  nature  of  the  product  as  depicted  in  the  various  levels 
of  the  Work  Breakdown  Structure  (product  risk). 

1.3.4  Cost  Risk 

This  is  the  risk  associated  with  the  ability  of  the  program  to  achieve  its  life-cycle  cost 
objectives.  Two  risk  areas  bearing  on  cost  are  (1)  the  risk  that  the  cost  estimates  and 
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objectives  are  accurate  and  reasonable  and  (2)  the  risk  that  program  execution  will  not 
meet  the  cost  objectives  as  a  result  of  a  failure  to  mitigate  technical  risks. 


1.3.5  Schedule  Risk 

These  risks  are  those  associated  with  the  adequacy  of  the  time  estimated  and  allocated  for 
the  development,  production,  and  fielding  of  the  system.  Two  risk  areas  bearing  on  sched¬ 
ule  risk  are  (1)  the  risk  that  the  schedule  estimates  and  objectives  are  realistic  and  reason¬ 
able  and  (2)  the  risk  that  program  execution  will  fall  short  of  the  schedule  objectives  as  a 
result  of  failure  to  mitigate  technical  risks. 


1.3.6  Risk  Ratings 

This  is  the  value  that  is  given  to  a  risk  event  (or  the  program  overall)  based  on  the  analysis 
of  the  likelihood /probability  and  consequences  of  the  event.  For  the  XYZ  program,  risk 
ratings  of  Low,  Moderate,  or  High  will  be  assigned  based  on  the  following  criteria.  See 
Section  3.3.2  for  guidance  on  determining  likelihood  and  consequences.  When  rating 
process  variance  from  best  practices,  there  is  no  rating  of  likelihood /probability,  rather 
the  level  would  be  a  measure  of  the  variance  from  best  practices  (see  Paragraph  3.3.2.3). 

•  Low  Risk:  Has  little  or  no  potential  for  increase  in  cost,  disruption  of  schedule,  or 
degradation  of  performance.  Actions  within  the  scope  of  the  planned  program  and  nor¬ 
mal  management  attention  should  result  in  controlling  acceptable  risk. 

•  Moderate  Risk:  May  cause  some  increase  in  cost,  disruption  of  schedule,  or  degra¬ 
dation  of  performance.  Special  action  and  management  attention  may  be  required  to 
control  acceptable  risk. 

•  High  Risk:  Likely  to  cause  significant  increase  in  cost,  disruption  of  schedule,  or 
degradation  of  performance.  Significant  additional  action  and  high  priority  manage¬ 
ment  attention  will  be  required  to  control  acceptable  risk. 

1.3.7  Independent  Risk  Assessor 

An  independent  risk  assessor  is  a  person  who  is  not  in  the  management  chain  or  directly 
involved  in  performing  the  tasks  being  assessed.  Use  of  independent  risk  assessors  is  a  valid 
technique  to  ensure  that  all  risk  areas  are  identified  and  that  the  consequence  and  likelihood/ 
probability  (or  process  variance)  are  properly  understood.  The  techmque  can  be  used  at 
different  program  levels,  e.g..  Program  Office,  Service  Field  Activities,  Contractors,  etc.  The 
Program  Manager  will  approve  the  use  of  independent  assessors,  as  needed. 

1.3.8  Templates  and  Best  Practices 

A  "template"  is  a  disciplined  approach  for  the  application  of  critical  engineering  and  manu¬ 
facturing  processes  that  are  essential  to  the  success  of  most  programs.  DoD  4245.7-M,  Transi¬ 
tion  from  Development  to  Production  Solving  the  Risk  Equation,  provides  a  number  of  such  tem¬ 
plates.  For  each  template  process  described  in  DoD  4245.7-M,  a  Best  Practice  Information  is 
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described  in  NAVSO  P-6071.  These  documents  outline  the  ideal  or  low  risk  approach  and 
thus  serve  as  a  baseline  from  which  risk  for  some  XYZ  processes  can  be  assessed. 

1.3.9  Metrics 

There  are  measures  used  to  indicate  progress  or  achievement. 

1.3.10  Critical  Program  Attributes 

Critical  Program  Attributes  are  performance,  cost,  and  schedule  properties  or  values  that  are 
vital  to  the  success  of  the  program.  They  are  derived  from  various  sources,  such  as  the  Acqui¬ 
sition  Program  Baseline,  exit  criteria  for  the  next  program  phase.  Key  Performance  Param¬ 
eters,  test  plans,  the  judgment  of  program  experts,  etc.  The  XYZ  program  will  track  these 
attributes  to  determine  the  progress  in  achieving  the  final  required  value.  See  Annex  A  for  a 
list  of  the  XYZ  Critical  Program  Attributes. 

2.0  RISK  MANAGEMENT  APPROACH 
2.1  GENERAL  APPROACH  AND  STATUS 

DoD  Directive  5000.1  states:  "Risks  must  be  well  understood,  and  risk  management  ap¬ 
proaches  developed,  before  decision  authorities  can  authorize  a  program  to  proceed  into 
the  next  phase  of  the  acquisition  process."  This  policy  is  implemented  in  DoD  Regulation 
5000.2-R,  with  more  detailed  guidance  provided  in  the  individual  Service  regulation.  The 
Defense  Acquisition  Deskbook  (Section  2.5.2)  provides  additional  guidance,  advice,  and 
wisdom  on  the  management  of  risk.  Figure  2-1  shows  how  the  XYZ  program  risk  man¬ 
agement  fits  into  the  phases  and  milestones  of  the  acquisition  process. 


»  CURRENT  STATUS 
•  BASELINE 
•  COST 
>  SCHEDULE 
PERFORMANCE 
EXECUTION  STATUS 


PLANS 

•  PROGRAM  PLANS 

•  EXIT  CRITERIA 


ASSESSMENT 

•  COST 

•  SCHEDULE 

•  PERFORMANCE 


Figure  2-1.  Risk  Management  and  the  Acquisition  Process 


B-7 


The  XYZ  program  will  use  a  centrally  developed  risk  management  strategy  throughout 
the  acquisition  process  and  decentralized  risk  planning,  assessment,  handling,  and  moni¬ 
toring.  XYZ  risk  management  is  applicable  to  all  acquisition  functional  areas. 

The  results  of  the  Concept  Exploration  Phase  of  the  pro^am  identified  potential  risk 
events  and  the  Acquisition  Strategy  reflects  the  program's  risk-handling  approach.  Over¬ 
all,  the  risk  of  the  XYZ  program  for  Milestone  I  was  assessed  as  moderate,  but  accept¬ 
able.  Moderate  risk  functional  areas  were  threat,  manufacturing,  cost,  funding,  and 
schedule.  The  remaining  functional  areas  of  technology,  design  and  engineering  (hard¬ 
ware  and  software),  support,  (schedule)  concurrency,  human  systems  integration,  and 
environmental  impact  were  assessed  as  low  risk. 


2.2  RISK  MANAGEMENT  STRATEGY 

The  basic  risk  management  strategy  is  intended  to  identify  critical  areas  and  risk  events, 
both  technical  and  non-technical,  and  take  necessary  action  to  handle  them  before  they 
can  become  problems,  causing  serious  cost,  schedule,  or  ^rformance  impacts.  This 
program  will  make  extensive  use  of  modeling  and  simulation,  technology  demonstra¬ 
tions,  and  prototype  testing  to  handle  risk. 

Risk  management  will  be  accomplished  using  the  integrated  Government-Contractor 
IPT  organization.  These  IPTs  will  use  a  structured  assessment  approach  to  identify  and 
analyze  those  processes  and  products  that  are  critical  to  meeting  the  program  objectives. 
They  will  then  develop  risk-handhng  options  to  mitigate  the  risks  and  monitor  the  effec¬ 
tiveness  of  the  selected  handling  options.  Key  to  the  success  of  the  risk  management 
effort  is  the  identification  of  the  resources  required  to  implement  the  developed  risk- 
handhng  options. 

Risk  information  will  be  captured  by  the  IPTs  in  a  risk  management  information  system 
(RMIS)  using  a  standard  Risk  Information  Form  (RIF).  The  RMIS  will  provide  standard 
reports,  and  is  capable  of  preparing  ad  hoc  tailored  reports.  See  Annex  B  for  a  descrip¬ 
tion  of  the  RMIS  and  RIF. 

Risk  information  will  be  included  in  all  program  reviews,  and  as  new  information  be¬ 
comes  available,  the  PMO  and  contractor  wiU  conduct  additional  reviews  to  ascertain  if 
new  risks  exist.  The  goal  is  to  be  continuously  looking  to  the  future  for  areas  that  may 
severely  impact  the  program. 


2.3  ORGANIZATION 

The  risk  organization  for  the  XYZ  program  is  shown  in  Figure  2-2.  This  is  not  a  separate 
organization,  but  rather  shows  how  risk  is  integrated  into  the  program's  existing  organi¬ 
zation  and  shows  risk  relationships  among  members  of  the  program  team. 
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Figure  2-2.  XYZ  Risk  Management  Organization 


2.3.1  Risk  Management  Coordinator 

The  Risk  Management  Coordinator,  the  XYZ  Technology  Assessment  and  R&D  Manager, 
is  overall  coordinator  of  the  Risk  Management  Program.  The  Risk  Management  Coordi¬ 
nator  is  responsible  for: 

•  Maintaining  this  Risk  Management  Plan 

•  Maintaining  the  Risk  Management  Data  Base 

•  Briefing  the  PM  on  the  status  of  XYZ  program  risk 

•  Tracking  efforts  to  reduce  moderate  and  high  risk  to  acceptable  levels 

•  Providing  risk  management  training 

•  Facilitating  risk  assessments  and 

•  Preparing  risk  briefings,  reports,  and  documents  required  for  Program  Reviews  and 
the  acquisition  Milestone  decision  processes. 

2.3.2  Program  Integrating  Integrated  Produ 
ctTeam(PIIPT) 

The  PIIPT  is  responsible  for  complying  with  the  DoD  risk  management  policy  and  for 
structuring  an  efficient  and  useful  XYZ  risk  management  approach.  The  Program  Man¬ 
ager  is  the  Chair  of  the  PIIPT.  The  PIIPT  membership  may  be  adjusted  but  is  initially 
established  as  the  chairs  of  the  Program  IPTs,  designated  sub-tier  IPTs,  and  the  Heads  of 
PMO  Functional  Offices. 
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2.3.3  PIPTs 

The  program  IPTs  are  responsible  for  implementing  risk  management  tasks  per  this  plan. 
This  includes  the  following  responsibilities: 

•  Review  and  recommend  to  the  Risk  Management  Coordinator  changes  on  the  overall 
risk  management  approach  based  on  lessons  learned. 

•  Quarterly,  or  as  directed,  update  the  program  risk  assessments  made  during  Phase  I. 

•  Review  and  be  prepared  to  justify  the  risk  assessments  made  and  the  risk  mitigation 
plans  proposed. 

•  Report  risk  to  the  Risk  Management  Coordinator  via  Risk  Information  Forms  (RIFs). 

•  Ensure  that  risk  is  a  consideration  at  each  Program  and  Design  Review. 

•  Ensure  Design/Build  Team  responsibilities  incorporate  appropriate  risk  management 
tasks. 


2.3.4  XYZ  Independent  Risk  Assessors 

Independent  Assessors  made  a  significant  contribution  to  the  XYZ  Milestone  I  risk  assess¬ 
ments.  The  use  of  independent  assessments  as  a  means  of  ensuring  that  all  risk  areas  are 
identified  will  continue,  when  necessary. 

2.3.5  Other  Risk  Assessment  Responsibilities 

The  Risk  Assessment  responsibilities  of  other  Systems  Command  codes.  Service  Field 
Activities,  Design/Build  Teams,  and  Contractors  will  be  as  described  in  Memoranda  of 
Agreement  (MOAs),  Memoranda  of  Understanding  (MOUs),  Systems  Command  Task¬ 
ing,  or  contracts.  This  RMP  should  be  used  as  a  guide  for  XYZ  risk  management  efforts. 

2.3.6  User  Participation 

The  Requirements  Organization  (specific  code)  is  the  focal  point  for  providing  the  Pro¬ 
gram  Executive  Officer  or  the  Project  Manager  with  user  identified  risk  assessments. 

2.3.7  Risk  Training 

The  key  to  the  success  of  the  risk  efforts  is  the  degree  to  which  all  members  of  the  team, 
both  Government  and  contractor  are  properly  trained.  The  XYZ  Program  Office  will 
provide  risk  training,  or  assign  members  to  training  classes,  during  acquisition  Phases  I 
and  n.  Key  personnel  with  XYZ  management  or  assessment  responsibilities  are  required 
to  attend.  All  members  of  the  team  will  receive,  at  a  minimum,  basic  risk  management 
training.  XYZ  sponsored  training  is  planned  to  be  presented  according  to  the  schedule 
provided  in  Annex  x  (not  provided). 
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3.0  RISK  MANAGEMENT  PROCESS  AND  PROCEDURES 


3.1  OVERVIEW 

This  section  describes  XYZ  program's  risk  management  process  and  provides  an  over¬ 
view  of  the  XYZ  risk  management  approach.  The  Defense  Acquisition  Deskbook  defines 
risk  management  as  "the  act  or  practice  of  controlling  risk.  It  includes  risk  planning, 
assessing  risk  areas,  developing  risk-handling  options,  monitoring  risks  to  determine  how 
risks  have  changed,  and  documenting  the  overall  risk  management  program."  Figure 
3-1  shows,  in  general  terms,  the  overall  risk  management  process  that  will  be  followed  in 
the  XYZ  program.  This  process  follows  DoD  and  Service  policies  and  guidelines  and 
incorporates  ideas  found  in  other  sources.  Each  of  the  risk  management  functions  shown 
in  Figure  3-1  is  discussed  in  the  following  paragraphs,  along  with  specific  procedures  for 
executing  them. 


Figure  3-1.  The  Risk  Management  Process 


3.2  RISK  PLANNING 

3.2.1  Process 

Risk  planning  consists  of  the  up-front  activities  necessary  to  execute  a  successful  risk 
management  program.  It  is  an  integral  part  of  normal  program  planning  and  manage¬ 
ment.  The  planning  should  address  each  of  the  other  risk  management  functions,  result¬ 
ing  in  an  organized  and  thorough  approach  to  assess,  handle,  and  monitor  risks.  It  should 
also  assign  responsibilities  for  specific  risk  management  actions  and  establish  risk  report¬ 
ing  and  documentation  requirements.  This  RMP  serves  as  the  basis  for  all  detailed  risk 
planning,  which  must  be  continuous. 

3.2.2  Procedures 

3.2.2.1  Responsibilities.  Each  IPT  is  responsible  for  conducting  risk  planning,  using  this 
RMP  as  the  basis.  The  planning  will  cover  all  aspects  of  risk  management  to  include 
assessment,  handling  options,  and  monitoring  of  risk  mitigation  activities.  The  Program 
Risk  Management  Coordinator  will  monitor  the  planning  activities  of  the  BPTs  to  ensure 
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that  they  are  consistent  with  this  RMP  and  that  appropriate  revisions  to  this  plan  are 
made  when  required  to  reflect  significant  changes  resulting  from  the  IPX  planning  efforts. 

Each  person  involved  in  the  design,  production,  operation,  support,  and  eventual  dis¬ 
posal  of  the  XYZ  system  or  any  of  its  systems  or  components  is  a  part  of  the  risk  manage¬ 
ment  process.  This  involvement  is  continuous  and  should  be  considered  a  part  of  the 
normal  management  process. 

3.2.2.2  Resources  and  Training.  An  effective  risk  management  program  requires  re¬ 
sources.  As  part  of  its  planning  process,  each  IPT  will  identify  the  resources  required  to 
implement  the  risk  management  actions.  These  resources  include  time,  material,  person¬ 
nel,  and  cost.  Training  is  major  consideration.  All  IPT  members  should  receive  instruc¬ 
tion  on  the  fundamentals  of  risk  management  and  special  training  in  their  area  of  respon¬ 
sibility,  if  necessary. 

3.2.2.3  Documentation  and  Reporting.  This  RMP  establishes  the  basic  documentation 
and  reporting  requirements  for  the  program.  IPTs  should  identify  any  additional  re¬ 
quirements  that  might  be  needed  to  effectively  manage  risk  at  their  level.  Any  such  addi¬ 
tional  requirements  must  not  conflict  with  the  basic  requirements  in  this  RMP. 

3.2.2.4  Metrics.  Each  IPT  should  establish  metrics  that  will  measure  the  effectiveness  of 
their  planned  risk-handling  options.  See  Annex  C  for  an  example  of  metrics  that  may  be 
used. 

3.2.2.5  Risk  Planning  Tools.  The  following  tools  can  be  useful  in  risk  planning.  It  may  be 
useful  to  provide  this  information  to  the  contractors  to  help  them  understand  the  XYZ 
program's  approach  to  managing  risk.  This  list  is  not  meant  to  be  exclusive. 

•  DoD  Manual  4245.7-M,  a  DoD  guide  for  assessing  process  technical  risk. 

•  The  Navy's  Best  Practices  Manual,  NAVSO  P-6071,  provides  additional  insight  into 
each  of  the  Templates  in  DoD  4245.7-M  and  a  checklist  for  each  template. 

•  Program  Manager's  Work  Station  (PMWS)  software,  may  be  useful  to  some  risk  as¬ 
sessors.  PMWS  has  a  Risk  Assessment  module  based  on  the  Template  Manual  and  Best 
Practices  Manual. 

•  Commercial  risk  management  software  offer  options. 

•  Government  risk  management  software,  such  as  Risk  Matrix  developed  by  Mitre 
Corporation  for  the  Air  Force  and  the  New  Attack  Submarine's  On-Line  Risk  Data  Base 
(OLRDB)  are  excellent  tools. 

3.2.2.6  Plan  Update.  This  RMP  wiU  be  updated,  if  necessary,  on  the  following  occasions: 
(1)  whenever  the  acquisition  strategy  changes,  or  there  is  a  major  change  in  program 
emphasis;  (2)  in  preparation  for  major  decision  points;  (3)  in  preparation  for  and  immedi¬ 
ately  following  technical  audits  and  reviews;  (4)  concurrent  with  the  review  and  update 
of  other  program  plans;  and  (5)  in  preparation  for  a  POM  submission. 
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3.3  RISK  ASSESSMENT 


The  risk  assessment  process  includes  the  identification  of  critical  risk  events/processes, 
which  could  have  an  adverse  impact  on  the  program,  and  the  analyses  of  these  events/ 
processes  to  determine  the  likelihood  of  occurrence/process  variance  and  consequences. 
It  is  the  most  demanding  and  time-consuming  activity  in  the  risk  management  process. 

3.3.1  Process 

3.3.1.1  Identification.  Risk  identification  is  the  first  step  in  the  assessment  process.  The 
basic  process  involves  searching  through  the  entire  XYZ  program  to  determine  those  criti¬ 
cal  events  that  would  prevent  the  program  from  achieving  its  objectives.  All  identified 
risks  will  be  documented  in  the  RMIS,  with  a  statement  of  the  risk  and  a  description  of  the 
conditions  or  situations  causing  concern  and  the  context  of  the  risk. 

Risks  will  be  identified  by  all  IPTs  and  by  any  individual  in  the  program.  The  lower-level 
IPTs  can  identify  significant  concerns  earlier  than  otherwise  might  be  the  case  and  iden¬ 
tify  those  events  in  critical  areas  that  must  be  dealt  with  to  avoid  adverse  consequences. 
Likewise,  individuals  involved  in  the  detailed  and  day-to-day  technical,  cost,  and  sched¬ 
uling  aspects  of  the  program  are  most  aware  of  the  potential  problems  (risks)  that  need  to 
be  managed. 

3.3.1.2  Analysis.  This  process  involves: 

•  Identification  of  WBS  elements 

•  Evaluation  of  the  WBS  elements  using  the  risk  areas  to  determine  risk  events 

•  Assignment  of  likelihood /probability  and  consequence  to  each  risk  event  to  estab¬ 
lish  a  risk  rating 

•  Prioritization  of  each  risk  event  relative  to  other  risks. 

Risk  analysis  should  be  supported  by  a  study,  test  results,  modeling  and  simulation,  trade 
study,  the  opinion  of  a  qualified  expert  (to  include  justification  of  his  or  her  judgment),  or 
any  other  accepted  analysis  technique.  The  DoD  Acquisition  Deskbook,  Action  2524.2 
describes  a  number  of  analysis  techniques  that  may  be  useful.  Evaluators  should  identify 
all  assumptions  made  in  assessing  risk.  When  appropriate,  a  sensitivity  analysis  should 
be  done  on  assumptions. 

Systems  engineering  analysis,  risk  assessments,  and  manpower  risk  assessments  provide 
additional  information  that  must  be  considered.  This  includes,  among  other  things,  envi¬ 
ronmental  impact,  system  safety  and  health  analysis,  and  security  considerations.  Classi¬ 
fied  programs  may  experience  difficulties  in  access,  facilities,  and  visitor  control  that  can 
introduce  risk  and  must  be  considered. 

The  analysis  of  individual  risk  will  be  the  responsibihty  of  the  IPT  identifying  the  risk,  or 
the  IPT  to  which  the  risk  has  been  assigned.  They  may  use  external  resources  for  assis¬ 
tance,  such  as  field  activities.  Service  laboratories,  and  contractors.  The  results  of  the 
analysis  of  all  identified  risks  must  be  documented  in  the  RMIS. 
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3.3.2  Procedures 

3.3.2.1  Assessments— General.  Risk  assessment  is  an  iterative  process,  with  each  assess¬ 
ment  building  on  the  results  of  previous  assessments.  The  current  baseline  assessment  is 
a  combination  of  the  risk  assessment  delivered  by  the  contractors  as  part  of  Phase  0,  the 
program  office  process  risk  assessment  done  before  Milestone  I,  and  the  post-award  Inte¬ 
grated  Baseline  Review  (IBR). 

For  the  program  office,  unless  otherwise  directed  in  individual  tasking,  program  level 
risk  assessments  will  be  presented  at  each  Program  Review  meeting  with  a  final  update 
not  later  than  6  months  before  the  next  scheduled  Milestone  decision.  The  primary  source 
of  information  for  the  next  assessment  will  be  the  current  assessment  baseline,  and  exist¬ 
ing  documentation  such  as.  Phase  0  study  results,  the  design  mission  profile  that  was 
done  as  part  of  Phase  0,  the  Integrated  Baseline  Review  (IBR),  which  will  be  conducted 
immediately  after  Phase  I  contract  award,  the  contract  WBS  that  is  part  of  the  IBR,  indus¬ 
try  best  practices  as  described  in  the  PMWS  Knowledgebase,  the  ORD,  the  Acquisition 
Program  Baseline  (APB),  and  any  contractor  design  documents. 

IPTs  should  continually  assess  the  risks  in  their  areas,  reviewing  risk-mitigation  actions 
and  the  critical  risk  areas  whenever  necessary  to  assess  progress.  For  contractors,  risk 
assessment  updates  should  be  made  as  necessary. 

The  risk  assessment  process  is  intended  to  be  flexible  enough  so  that  field  activities,  ser¬ 
vice  laboratories,  and  contractors  may  use  their  judgment  in  structuring  procedures  con¬ 
sidered  most  successful  in  identifying  and  analyzing  all  risk  areas. 

3.3.2.2  Identification.  Following  is  a  description  of  step-by-step  procedures  that  evalua¬ 
tors  may  use  as  a  guide  to  identify  program  risks. 

•  Step  One — ^Understand  the  requirements  and  the  program  performance  goals,  which 
are  defined  as  thresholds  and  objectives  (see  5000.2-R).  Describe  the  operational  (func¬ 
tional  and  environmental)  conditions  under  which  the  values  must  be  achieved  by  refer¬ 
ring  or  relating  to  design  documents.  The  ORD  and  APB  contain  Key  Performance  Pa¬ 
rameters  (KPPs). 

•  Step  Two — ^Determine  the  engineering  and  manufacturing  processes  that  are  needed 
to  design,  develop,  produce,  and  support  the  system.  Obtain  industry  best  practices  for 
these  processes. 

•  Step  Three — ^Identify  contract  WBS  elements  (to  include  products  and  processes). 

•  Step  Four — ^Evaluate  each  WBS  element  against  sources/ areas  of  risk  described  in 
Table  4-2  fo  the  DSMC  Risk  Management  Guide. 

•  Step  Five — ^Assign  a  probability  and  consequence  to  each  risk  event 

•  Step  Six — ^Prioritize  the  risk  events. 

Following  are  indicators  that  IPTs  may  find  helpful  in  identifying  and  assessing  risk: 
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•  Lack  of  Stability,  Clarity,  or  LFnderstanding  of  Requirements:  Requirements  drive 
the  design  of  the  system.  Changing  or  poorly  stated  requirements  guarantees  the  intro¬ 
duction  of  performance,  cost,  and  schedule  problems. 

•  Failure  to  Use  Best  Practices  virtually  assures  that  the  program  will  experience  some 
risk.  The  further  a  contractor  deviates  from  best  practices,  the  higher  the  risk. 

•  New  Processes  should  always  be  suspect,  whether  they  are  related  to  design,  analy¬ 
sis,  or  production.  Until  they  are  vahdated,  and  until  the  people  who  implement  them 
have  been  trained  and  have  experience  in  successfully  using  the  process,  there  is  risk. 

•  Any  Process  Lacking  Rigor  should  also  be  suspect;  it  is  inherently  risky.  To  have 
rigor,  a  process  should  be  mature  and  documented,  it  should  have  been  validated,  and  it 
should  be  strictly  followed. 

•  Insufficient  Resources:  People,  funds,  schedule,  and  tools  are  necessary  ingredients 
for  successfully  implementing  a  process.  If  any  are  inadequate,  to  include  the  qualifica¬ 
tions  of  the  people,  there  is  risk. 

•  Test  Failure  may  indicate  corrective  action  is  necessary.  Some  corrective  actions  may 
not  fit  available  resources,  or  the  schedule,  and  (for  other  reasons  as  well)  may  contain 
risk. 

•  Qualified  Supplier  Availability:  A  supplier  not  experienced  with  the  processes  for 
designing  and  producing  a  specific  product  is  not  a  qualified  supplier  and  is  a  source  of  risk. 

•  Negative  Trends  or  Forecasts  are  cause  for  concern  (risk)  and  may  require  specific 
actions  to  turn  around. 

There  are  a  number  of  techniques  and  tools  available  for  identifying  risks.  Among  them 
are: 

•  Best  Judgment:  The  knowledge  and  experience  of  the  collective,  multi-disciplined 
Integrated  Project  Team  (IPT)  members  and  the  opinion  of  subject  matter  experts  (SMEs) 
are  the  most  common  source  of  risk  identification. 

•  Lessons  Learned  from  similar  processes  can  serve  as  a  basehne  for  the  successful 
way  to  achieve  requirements.  If  there  is  a  departure  from  the  successful  way,  there  may 
be  risk. 

•  DoD  4245.7-M,  "Transition  from  Development  to  Production,"  is  often  called  the 
"Templates"  book  because  it  identifies  technical  risk  areas  and  provides,  in  "bullet"  form, 
suggestions  for  avoiding  those  risks.  It  focuses  on  the  technical  details  of  product  design, 
test,  and  production  to  help  managers  proactively  manage  risk.  It  also  includes  chapters 
on  Facihties,  Logistics,  and  Management,  which  make  this  a  useful  tool  in  identifying 
weak  areas  of  XYZ  planned  processes  early  enough  to  implement  actions  needed  to  avoid 
adverse  consequences. 
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•  The  NAVSO  P-6071  Best  Practices  Manual  was  developed  by  the  Navy  to  add  depth 
to  the  Template  Book,  DoD  4245.7-M. 

•  Critical  Program  Attributes  are  metrics  that  the  program  office  developed  to  mea¬ 
sure  progress  toward  meeting  our  objectives.  Team  members,  IPTs,  functional  managers, 
contractors,  etc.,  may  develop  their  own  metrics  to  support  these  measurements.  The 
attributes  may  be  specification  requirements,  contract  requirements,  or  measurable  pa 

rameters  from  any  agreement  or  tasking.  The  idea  is  to  provide  a  means  to  measure 
whether  we  are  on  track  in  achieving  our  objectives. 

•  Methods  and  Metrics  for  Product  Success  is  a  manual  published  by  the  Office  of  the 
Assistant  Secretary  of  the  Navy  (RDA)  Product  Integrity  Directorate.  It  highlights  areas 
related  to  design,  test,  and  production  processes  where  problems  are  most  often  found 
and  metrics  for  the  measurement  of  effectiveness  of  the  processes.  It  also  describes  the 
software  tool.  Program  Manager's  Work  Station  (PMWS).  See  next  paragraph. 

•  PMWS  contains  risk  management  software,  "Technical  Risk  Identification  and  Miti¬ 
gation  System  (TRIMS)  and  Knowledgebase."  They  provide  a  tailorable  management 
system  based  on  NAVSO  P-6071  and  DpD  4245.7-M.  The  PMWS  provides  a  compact  disk 
(CD)  that  contains  the  necessary  programs  for  assessing  a  program's  risk  and  software  for 
program  management.  PMWS  can  be  obtained  by  calling  the  Best  Manufacturing  Pro¬ 
gram  (BMP)  Office  at  (301)  403-8100. 

•  New  Nuclear  Submarine  (NSSN)  On-Line  Risk  Database  (ONLRB)  is  a  software 
tool  may  be  used  to  support  the  XYZ  Risk  Management  Process.  The  tool  helps  IPTs  in  the 
identification  and  assessment  of  risk  and  management  of  handling  efforts. 

•  Risk  Matrix  is  another  candidate  for  use  by  the  PMO.  It  is  an  automated  tool,  devel¬ 
oped  by  Mitre  Corporation,  that  supports  a  structured  approach  for  identifying  risk  and 
assessing  its  potential  program  impact.  It  is  especially  helpful  for  prioritizing  risks. 

•  Requirements  Documents  describe  the  output  of  our  efforts.  IPT  efforts  need  to  be 
monitored  continuously  to  ensure  requirements  are  met  on  time  and  within  budget.  When 
they  aren't,  there  is  risk. 

•  Contracting  for  Risk  Management  helps  ensure  the  people  involved  with  the  details 
of  the  technical  processes  of  design,  test,  and  production  are  involved  with  managing 
risk.  The  principle  here  is  that  those  performing  the  technical  details  are  normally  the 
first  ones  to  know  when  risks  exist. 

•  Quality  Standards,  such  as  IS09000,  ANSI/ASQC  Q  9000,  MIL-HDBK  9000,  and 
others  describe  processes  for  developing  and  producing  quality  products.  Comparing 
our  processes  with  these  standards  can  highlight  areas  we  may  want  to  change  to  avoid 
risk. 

•  Use  of  Independent  Risk  Assessors  is  a  method  to  help  ensure  all  risk  is  identified. 
The  knowledgeable,  experienced  people  are  independent  from  the  management  and  ex- 
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ecution  of  the  processes  and  procedures  being  reviewed.  Independent  assessment  pro¬ 
motes  questions  and  observations  not  otherwise  achievable. 

33.2.3  Analysis.  Risk  analysis  is  an  evaluation  of  the  identified  risk  events  to  determine 
possible  outcomes,  critical  process  variance  from  known  best  practices,  the  likelihood  of 
those  events  occurring,  and  the  consequences  of  the  outcomes.  Once  this  information  has 
been  determined,  the  risk  event  may  be  rated  against  the  program's  criteria  and  an  over¬ 
all  assessment  of  low,  moderate,  or  high  assigned.  Figure  3-2  depicts  the  risk  analysis 
process  and  procedures. 


Risk  Analysis  Process 


Figure  3-2.  Risk  Assessment  Process 


Critical  Process  Variance.  For  each  process  risk  related  event  identified,  the  variance  of 
the  process  from  known  standards  or  best  practices  must  be  determined.  As  shown  in 
Figure  3-2,  there  are  five  levels  (a-e)  in  the  XYZ  risk  assessment  process,  with  the  corre¬ 
sponding  criteria  of  Minimal,  Small,  Acceptable,  Large,  and  Significant.  If  there  is  no  vari¬ 
ance  then  there  is  no  risk. 

Likelihood/Probability.  For  each  risk  area  identified,  the  likelihood  the  risk  will  happen 
must  be  determined.  As  shown  in  Figure  3-2,  there  are  five  levels  (a-e)  in  the  )(YZ  risk  assess¬ 
ment  process,  with  the  corresponding  criteria  of  Remote,  Unlikely,  Likely,  Highly  Likely,  and 
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Near  Certainty.  If  there  is  zero  Ukelihood  of  an  event,  there  is  no  risk  per  our  definition. 

Consequence.  For  each  risk  area  identified,  the  following  question  must  be  answered: 
Given  the  event  occurs,  what  is  the  magnitude  of  the  consequence?  As  shown  in  the  figure,  there 
are  five  levels  of  consequence  (1-5).  "Consequence"  is  a  multifaceted  issue.  For  this 
program,  there  are  four  areas  that  we  will  evaluate  when  determining  consequence:  tech¬ 
nical  performance,  schedule,  cost,  and  impact  on  other  teams.  At  least  one  of  the  four 
consequence  areas  needs  to  apply  for  there  to  be  risk;  if  there  is  no  adverse  consequence  in 
any  of  the  areas,  there  is  no  risk. 

•  Technical  Performance:  This  category  includes  all  requirements  that  are  not  included 
in  the  other  three  metrics  of  the  Consequence  table.  The  wording  of  each  level  is  oriented 
toward  design  processes,  production  processes,  life  cycle  support,  and  to  retirement  of 
the  system.  For  example,  the  word  "margin"  could  apply  to  weight  margin  during  de¬ 
sign,  safety  margin  during  testing,  or  machine  performance  margin  during  production. 

•  Schedule:  The  words  used  in  the  Schedule  column,  as  in  all  columns  of  the  Conse¬ 
quence  table,  are  meant  to  be  universally  applied.  Avoid  excluding  a  consequence  level 
from  consideration  just  because  it  doesn't  match  your  team's  specific  definitions.  In  other 
words,  phrases  such  as  need  dates,  key  milestones,  critical  path,  and  key  team  milestones 
are  meant  to  apply  to  all  IPTs. 

•  Cost:  Since  costs  vary  from  component  to  component  and  process  to  process,  the 
percentage  criteria  shown  in  the  figure  may  not  strictly  apply  at  the  lower  levels  of  the 
WBS.  These  team  leaders  can  set  the  percentage  criteria  that  best  reflects  their  situation. 
However,  when  costs  are  rolled  up  at  higher  levels  (e.g..  Program),  the  following  defini¬ 
tions  will  be  used:  Level  1 — no  change.  Level  2 — <5%,  Level  3 — ^5-7%,  Level  4 — 7-10%, 
and  Level  5 — >10%. 

•  Impact  on  Other  Teams:  Both  the  consequence  of  a  risk  and  the  mitigation  actions 
associated  with  reducing  the  risk  may  impact  another  team.  This  may  involve  additional 
coordination  or  management  attention  (resources)  and  may  therefore  increase  the  level  of 
risk.  This  is  especially  true  of  common  technical  processes. 

Risk  Rating.  Probability  and  consequence  should  not  always  be  considered  equally;  for 
example,  there  may  be  consequences  so  severe  that  it  is  considered  high  risk  even  though 
the  probability  to  achieve  a  particular  outcome  is  low.  After  deciding  a  level  of  process 
variance/likelihood  (a  through  e)  and  a  level  of  consequence  (1  through  5),  enter  the 
Assessment  Guide  portion  of  Figure  3-2  to  obtain  a  risk  rating  (green  =  LOW,  yellow  = 
MOD,  and  red  =  HIGH).  For  example;  consequence/process  variance/likelihood  level 
2b  corresponds  to  LOW  risk,  level  3d  corresponds  to  MOD  risk,  level  4c  corresponds  to 
HIGH  risk.  After  obtaining  the  risk  rating,  make  a  subjective  comparison  of  the  risk  event 
with  the  applicable  rating  definition  in  Figure  3-2  (e.g.,  High=unacceptable,  major  dis¬ 
ruptions,  etc.).  There  should  be  a  close  match.  If  there  isn't,  consider  reevaluating  the 
level  of  likelihood  or  consequence.  Those  risk  events  that  are  assessed  as  moderate  or 
high  should  be  submitted  to  the  XYZ  Risk  Management  Coordinator  on  a  RIF. 
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Figure  3-2  is  useful  to  convey  information  to  decision  makers  and  wiU  be  used  primarily  for 
that  purpose.  The  PMO  wUl  use  the  Risk  Tracking  Report  and  Watchlist.  (See  Annex  D.) 

3.4  RISK  HANDLING 

3.4.1  Process 

After  the  program's  risks  have  been  identified  and  assessed,  the  approach  to  handling 
each  significant  risk  must  be  developed.  There  are  essentially  four  techniques  or  options 
for  handling  risks:  avoidance,  control,  transfer,  and  assumption.  For  all  identified  risks, 
the  various  handling  techniques  should  be  evaluated  in  terms  of  feasibility,  expected  ef¬ 
fectiveness,  cost  and  schedule  implications,  and  the  effect  on  the  system's  technical  per¬ 
formance,  and  the  most  suitable  technique  selected.  Section  2524.3  of  the  DoD  Acquisi¬ 
tion  Deskbook  contains  information  on  the  risk-handling  techniques  and  various  actions 
that  can  be  used  to  implement  them.  The  results  of  the  evaluation  and  selection  will  be 
included  and  documented  in  the  RMIS  using  the  RTF.  This  documentation  will  include: 
what  must  be  done,  the  level  of  effort  and  materials  required,  the  estimated  cost  to  imple¬ 
ment  the  plan,  a  proposed  schedule  showing  the  proposed  start  date,  the  time  phasing  of 
significant  risk  reduction  activities,  the  completion  date,  and  their  relationship  to  signifi¬ 
cant  Program  activities/milestones  (an  example  is  provided  in  Annex  B),  recommended 
metrics  for  tracking  the  action,  a  list  of  all  assumptions,  and  the  person  responsible  for 
implementing  and  tracking  the  selected  option. 

3.4.2  Procedures 

The  IPT  that  assessed  the  risk  is  responsible  for  evaluating  and  recommending  to  the  PM 
the  risk-handling  options  that  are  best  fitted  to  the  program's  circumstances.  Once  ap¬ 
proved,  these  are  included  in  the  program's  acquisition  strategy  or  management  plans,  as 
appropriate. 

For  each  selected  handling  option,  the  responsible  IPT  will  develop  specific  tasks  that,  when 
implemented,  will  handle  the  risk.  The  task  descriptions  should  explain  what  has  to  be  done, 
the  level  of  effort,  and  identify  necessary  resources.  It  should  also  provide  a  proposed  sched¬ 
ule  to  accomplish  the  actions  including  the  start  date,  the  time  phasing  of  significant  risk 
reduction  activities,  the  completion  date,  and  their  relationship  to  significant  Program  activi¬ 
ties/milestones  (an  example  is  provided  in  Annex  B),  and  a  cost  estimate.  The  description  of 
the  handling  options  should  list  all  assumptions  used  in  the  development  of  the  handling 
tasks.  Assumptions  should  be  included  in  the  RIF.  Recommended  actions  that  require  re¬ 
sources  outside  the  scope  of  a  contract  or  official  tasking  should  be  clearly  identified,  and  the 
IPTs,  the  risk  area,  or  other  handling  plans  that  may  be  impacted  should  be  listed. 

Reducing  requirements  as  a  risk  avoidance  technique  will  be  used  only  as  a  last  resort, 
and  then  only  with  the  participation  and  approval  of  the  user's  representative. 

DoD  4245.7-M  Templates  and  NAVSO  P-6071  Best  Practices,  are  useful  in  developing 
risk-handling  actions  for  design,  test,  or  manufacturing  process  risks. 
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3.5  RISK  MONITORING 

3.5.1  Process 

Risk  monitoring  systematically  tracks  and  evaluates  the  performance  of  risk-handling 
actions.  It  is  part  of  the  PMO  Wction  and  responsibility  and  will  not  become  a  separate 
discipline.  Essentially,  it  compares  predicted  results  of  planned  actions  with  the  results 
actually  achieved  to  determine  status  and  the  need  for  any  change  in  risk-handling  ac¬ 
tions.  The  effectiveness  of  the  risk-monitoring  process  depends  on  the  establishment  of  a 
management  indicator  system  (metrics)  that  provides  accurate,  timely,  and  relevant  risk 
information  in  a  clear,  easily  understood  manner.  (See  Annex  D.)  The  metrics  selected  to 
monitor  program  status  must  adequately  portray  the  true  state  of  the  risk  events  and 
handling  actions.  Otherwise,  indicators  of  risks  that  are  about  to  become  problems  will 
go  undetected. 

To  ensure  that  significant  risks  are  effectively  monitored,  risk-handling  actions  (which 
include  specific  events,  schedules,  and  "success"  criteria)  will  be  reflected  in  integrated 
program  planning  and  scheduling.  Identifying  these  risk-handling  actions  and  events  in 
the  context  of  Work  Breakdown  Structure  (WBS)  elements  establishes  a  linkage  between 
them  and  specific  work  packages,  making  it  easier  to  determine  the  impact  of  actions  on 
cost,  schedule,  and  performance.  The  detailed  information  on  risk-handling  actions  and 
events  will  be  included  in  the  RIF  for  each  identified  risk,  and  thus  be  resident  in  the 
RMIS. 

3.5.2  Procedures 

The  functioning  of  IPTs  is  crucial  to  effective  risk  monitoring.  They  are  the  "front  line"  for 
obtaining  indications  that  risk-handling  efforts  are  achieving  their  desired  effects.  Each 
IPT  is  responsible  for  monitoring  and  reporting  the  effectiveness  of  the  handling  actions 
for  the  risks  assigned.  Overall  XYZ  program  risk  assessment  reports  will  be  prepared  by 
the  XYZ  Risk  Management  Coordinator  working  with  the  cognizant  IPT. 

Many  techniques  and  tools  are  available  for  monitoring  the  effectiveness  of  risk-handling 
actions,  and  IPTs  must  ensure  that  they  select  those  that  best  suit  their  needs.  No  single 
technique  or  tool  is  capable  of  providing  a  complete  answer — a  combination  must  be 
used.  At  a  minimum,  each  IPT  will  maintain  a  watchlist  of  identified  high  priority  risks. 
See  Section  2524.4  of  the  DoD  Acquisition  Deskbook  for  information  on  specific  tech¬ 
niques. 

Risks  rated  as  Moderate  or  High  risk  will  be  reported  to  the  XYZ  Risk  Management  Coor¬ 
dinator,  who  will  also  track  them,  using  information  provided  by  the  appropriate  IPT, 
until  the  risk  is  considered  Low  and  recommended  for  "Close  Out."  The  IPT  that  initially 
reported  the  risk  retains  ownership  and  cognizance  for  reporting  status  and  keeping  the 
database  current.  Ownership  means  implementing  handling  plans  and  providing  peri¬ 
odic  status  of  the  risk  and  of  the  handling  plans.  Risk  will  be  made  an  agenda  item  at 
each  management  or  design  review,  providing  an  opportunity  for  all  concerned  to  offer 
suggestions  for  the  best  approach  to  managing  risk.  Communicating  risk  increases  the 
program's  credibility  and  allows  early  actions  to  minimize  adverse  consequences. 
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The  risk  management  process  is  continuous.  Information  obtained  from  the  monitoring 
process  is  fed  back  for  reassessment  and  evaluations  of  handling  actions.  When  a  risk 
area  is  changed  to  Low,  it  is  put  into  a  "Historical  File"  by  the  Risk  Management  Coordi¬ 
nator  and  it  is  no  longer  tracked  by  the  XYZ  PMO.  The  "owners"  of  all  Low  risk  areais 
will  continue  monitoring  Low  risks  to  ensure  they  stay  Low. 

The  status  of  the  risks  and  the  effectiveness  of  the  risk-handling  actions  will  be  reported 
to  the  Risk  Management  Coordinator: 

•  Quarterly 

•  When  the  IPT  determines  that  the  status  of  the  risk  area  has  changed  significantly  (as 
a  minimum  when  the  risk  changes  from  high  to  moderate  to  low,  or  vice  versa) 

•  When  requested  by  the  Program  Manager. 

4.0  RISK  MANAGEMENT  INFORMATION  SYSTEM  AND  DOCUMENTATION 

The  XYZ  program  will  use  the  XXX  database  management  system  as  its  RMIS.  The  sys¬ 
tem  will  contain  all  of  the  information  necessary  to  satisfy  the  program  documentation 

and  reporting  requirements. 

4.1  RISK  MANAGEMENT  INFORMATION  SYSTEM  (RMIS) 

The  RMIS  stores  and  allows  retrieval  of  risk-related  data.  It  provides  data  for  creating 
reports  and  serves  as  the  repository  for  all  current  and  historical  information  related  to 
risk.  This  information  will  include  risk  assessment  documents,  contract  deliverables,  if 
appropriate,  and  any  other  risk-related  reports.  The  PMO  will  use  data  from  the  RMIS  to 
create  reports  for  senior  management  and  retrieve  data  for  day-to-day  management  of 
the  program.  The  program  produces  a  set  of  standard  reports  for  periodic  reporting  and 
has  the  ability  to  create  ad  hoc  reports  in  response  to  special  queries.  See  Annex  D  for  a 
detailed  discussion  of  the  RMIS. 

Data  are  entered  into  the  RMIS  using  the  Risk  Information  Form  (RIF).  The  RIF  gives 
members  of  the  project  team,  both  Government  and  contractors,  a  standard  format  for 
reporting  risk-related  information.  The  RIF  should  be  used  when  a  potential  risk  event  is 
identified  and  will  be  updated  as  information  becomes  available  as  the  assessment,  han¬ 
dling,  and  monitoring  functions  are  executed. 

4.2  RISK  DOCUMENTATION 

All  program  risk  management  information  will  be  documented,  using  the  RIF  as  the  stan¬ 
dard  RMIS  data  entry  form.  The  following  paragraphs  provide  guidance  on  documenta¬ 
tion  requirements  for  the  various  risk  management  functions. 
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4.2.1  Risk-Assessment  Documentation 

Risk  assessments  form  the  basis  for  many  program  decisions.  From  time  to  time,  the  PM 
will  need  a  detailed  report  of  any  assessment  of  a  risk  event.  It  is  critical  that  all  aspects  of 
the  risk  management  process  are  documented. 

4.2.2  Risk-Handling  Documentation 

Risk-handling  documentation  will  be  used  to  provide  the  PM  with  the  information  he 
needs  to  choose  the  preferred  mitigation  option. 

4.2.3  Risk-Monitoring  Documentation 

The  PM  needs  a  summary  document  that  tracks  the  status  of  high  and  moderate  risks. 
The  Risk  Management  Coordinator  will  produce  a  risk  tracking  list,  for  example,  that 
uses  information  that  has  been  entered  from  the  RMIS.  This  document  will  be  produced 
on  a  monthly  basis. 

4.3  REPORTS 

Reports  are  used  to  convey  information  to  decision-makers  and  team  members  on  the 
status  of  the  program  and  the  effectiveness  of  the  risk  management  program.  Every  effort 
will  be  made  to  generate  reports  using  the  data  resident  in  the  RMIS. 

4.3.1  Standard  Reports 

The  RMIS  will  have  a  set  of  standard  reports.  If  IPTs  or  functional  managers  need  addi¬ 
tional  reports,  they  should  work  with  the  Risk  Management  Coordinator  to  create  them. 
Access  to  the  reporting  system  will  be  controlled;  however,  any  member  of  the  Govern¬ 
ment  or  contractor  team  may  obtain  a  password  to  gain  access  to  the  information.  See 
Annex  B  for  a  description  of  the  XYZ  program  reports. 

4.3.2  Ad  Hoc  Reports 

In  addition  to  standard  reports,  the  PMO  will  need  to  create  ad  hoc  reports  in  response  to 
special  queries.  The  Risk  Management  Coordinator  will  be  responsible  for  these  reports. 
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ANNEX  A 

CRITICAL  PROGRAM  ATTRIBUTES 


Category 

Description 

Responsible  IPT 

Remarks 

Performance/Physlcal 

Speed 

Weight 

Endurance 

Crew  Size 

Survivability 

Maneuverability 

Size 

Receiver  Range 

Transmitter  Range 

Data  Link  Operations 

Recovery  Time 

Initial  Setup 

Identification  Time 

Accuracy  Location 

Probability  of  Accurate  ID 

Reliability 

Maintainability 

Availability 

Etc. 

Cost 

Operating  and  Support  Costs 

Etc, 

Processes 

Requirements  Stable 

Test  Plan  Approved 

Exit  Criteria 

Engine  Bench  Test 

Accuracy  Verified  by  Test  Data 
and  Analysis 

Toolproofing  Completed 

Logistics  Support  Reviewed  by 
User 
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ANNEX  B 

PROGRAM  RISK  REDUCTION  SCHEDULE 


m 


R 

I 

S 

K 

R 

A 

T 

I 

N 

G 


Accomplished  • 


Planned 


L 

O 

W 


CY 


Determine  and  flowdown  requirements,  evaluate  potential  hardware  and  software  solutions.  Gather  data 
on  NDI  capabilities,  limitations,  evaluate  alternatives  and  pick  lower  risk  solutions. 


Simulations  to  evaluate  subsystem  interactions,  timing  issues.  Simulations  to  evaluate  target  sets, 
'  environment  effects. 


Preliminary  design  and  trade  studies  to  work  issues  such  as  temperature  and  shock  environments. 
Develop  baseline  design.  Reassess  risk. 


I  Get  hardware  and  software  in  place  for  pre-EMD  simulations.  Consolidate  team  structure  and  supplier. 


Hardware-irhthe-Loop(HWIL)  and  performance  prediction  demo.  Supporting  analyses  and  design  studies. 


I 

k 

I 

t 

i 


Initiate  detailed  trade  studies  and  identify  alternatives.  Validate  and  implement  trade  study  decisions 
with  customer  on  IPD  teams  for  lower  risk  options.  Reassess  risk. 


n  Extensive  simulations  &  HWIL  testing.  Developmental  test  program,  supporting 
anflivaofi.  reviews  end  decisions. 


analyses,  reviews  and  decisions. 

1  Systems  integration  testing  (supported  by  continued  simulations)  to 
verify  design.  TAAF  program  with  selected  subsystems.  Reassess  risk. 


t 


I  Qualification  testing. 


MS  II 

V 


Operational  testing  & 
simulations. 


I 

X- 


Production. 


SRR  PSR 


CDR 


PRR 


MS  III 


PD&RR 

EMD 

1996 

1997 

1998 

1999 

2000 

2001 

2002 

2003 

XYZ  Program  Risk  Reduction  Scheduie  (Exampie) 
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ANNEX  C 

PROGRAM  METRIC  EXAMPLES 


Examples  of  Product^Related  Metrics 


Engineering 

Requirements 

Production 

Support 

Key  Design  Parameters 
Weight 

Size 

Endurance 

Range 

Design  Maturity 

Open  problem  reports 

Number  of  engineer¬ 
ing  change  proposals 

Number  of  drawings 
released 

Failure  activities 

Computer  Resource 
Utilization 

Etc. 

Requirements  Traceability 
Requirements  Stability 
Threat  Stability 

Design  Mission  Profile 

Manufacturing  Yields 
Incoming  Material  Yields 
Delinquent  Requisitions 
Unit  Production  Cost 
Process  Proofing 

Waste 

Personnel  Stability 

Special  Tools  and  Test 
Equipment 

Requirements 

Support  Infrastructure 
Footprint 

Manpower  Estimates 

Examples  of  Process  Metrics 


Design 

Requirements 

Trade 

Studies 

Design 

Process 

Integrated 
Test  Plan 

Failure 

Reporting 

System 

Manufacturing 

Plan 

Development 
of  require¬ 
ments  trace- 
ability  plan 

Development 
of  specification 
tree 

Specifications 
reviewed  for: 

Definition  of 
all  use  envi¬ 
ronments 

Definition  of 
all  functional 
requirements 
for  each 
mission 
performed 

Users  needs 
prioritized 

Alternative 
system  con¬ 
figurations 
selected 

Test  methods 
selected 

Design 

requirements 

stability 

Producibility 

analysis 

conducted 

Design 
analyzed  for: 

Cost 

Parts 

reduction 

Manufac¬ 

turability 

Testability 

All  develop¬ 
mental  tests  at 
system  and 
subsystem 
level  identified 

Identification  of 
who  will  do  test 
(Government, 
contractor, 
supplier) 

Contractor 
corporate-level 
management 
involved  in 
failure  reporting 
and  corrective 
action  process 

Responsibility 
for  analysis 
and  corrective 
action  assigned 
to  specific 
individual  with 
close-out  date 

Plan  documents 
methods  by 
which  design  to 
be  built 

Plan  contains 
sequence  and 
schedule  of 
events  at  con¬ 
tractor  and  sub¬ 
contractor  level 
that  defines  use 
of  materials, 
fabrication  flow, 
test  equipment, 
tools,  facilities, 
and  personnel 

Reflects  manu¬ 
facturing  inclu¬ 
sion  in  design 
process.  In¬ 
cludes  Identifi¬ 
cation  and 
assessment  of 
design  facilities 
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Examples  of  Cost  and  Schedule  Metrics 


Cost 

Schedule 

Cost  variance 

Cost  performance  index 
Estimate  at  completion 
Management  reserve 

Schedule  variance 

Schedule  performance  Index 

Design  Scheduie  Performance 
Manufacturing  Schedule  Performance 
Test  Schedule  Performance 
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ANNEX  D 

MANAGEMENT  INFORMATION  SYSTEM  AND  DOCUMENTATION 


1.0  DESCRIPTION 

In  order  to  manage  risk,  we  need  a  database  management  system  that  stores  and  allows 
retrieval  of  risk-related  data.  The  Risk  Management  Information  System  provides  data 
for  creating  reports  and  serves  as  the  repository  for  all  current  and  historical  information 
related  to  risk.  This  information  may  include  risk  assessment  documents,  contract 
deliverables,  if  appropriate,  and  any  other  risk-related  reports.  The  Risk  Management 
Coordinator  is  responsible  for  the  overall  maintenance  of  the  RMIS,  and  he  or  his  desig¬ 
nee  are  the  only  persons  who  may  enter  data  into  the  database. 

The  RMIS  will  have  a  set  of  standard  reports.  If  IPTs  or  functional  managers  need  addi¬ 
tional  reports,  they  should  work  with  the  Risk  Management  Coordinator  to  create  them. 
Access  to  the  reporting  system  will  be  controlled;  however,  any  member  of  the  Govern¬ 
ment  or  contractor  team  may  obtain  a  password  to  gain  access  to  the  information. 

In  addition  to  standard  reports,  the  PMO  will  need  to  create  ad  hoc  reports  in  response  to 
special  queries  etc.  The  Risk  Management  Coordinator  will  be  responsible  for  these  re¬ 
ports.  Figure  B-1  shows  a  concept  for  a  management  and  reporting  system. 


Figure  B-1.  Conceptual  Risk  Management  and  Reporting  System 
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2.0  RISK  MANAGEMENT  REPORTS 

The  following  are  examples  of  basic  reports  that  a  PMO  may  use  to  manage  its  risk  pro¬ 
gram.  Each  office  should  coordinate  with  the  Risk  Management  Coordinator  to  tailor  and 
amplify  them,  if  necessary,  to  meets  its  specific  needs. 

2.1  RISK  INFORMATION  FORM  (MANDATORY) 

The  PMO  needs  a  document  that  serves  the  dual  purpose  of  a  source  of  data  entry  infor¬ 
mation  and  a  report  of  basic  information  for  the  IPTs,  etc.  The  Risk  Information  Form 
(RIF)  serves  this  purpose.  It  gives  members  of  the  project  team,  both  Government  and 
contractors,  a  format  for  reporting  risk-related  information.  The  RIF  should  be  used  when 
a  potential  risk  event  is  identified  and  updated  over  time  as  information  becomes  avail¬ 
able  and  the  status  changes.  As  a  source  of  data  entry,  the  RIF  allows  the  database  admin¬ 
istrator  to  control  entries.  The  format  for  a  RIF  is  included  on  page  C-26. 

2.2  RISK  ASSESSMENT  REPORT  (NOT  MANDATORY) 

Risk  assessments  form  the  basis  for  many  program  decisions,  and  the  PM  may  need  a 
detailed  report  of  assessments  of  a  risk  event  that  has  been  done.  A  Risk  Assessment 
Report  (RAR)  is  prepared  by  the  team  that  assessed  a  risk  event  and  amplifies  the  infor¬ 
mation  in  the  RIF.  It  documents  the  identification,  analysis,  and  handling  processes  and 
results.  The  RAR  amplifies  the  summary  contained  in  the  RIF,  is  the  basis  for  developing 
risk-handling  plans,  and  serves  as  a  historical  recording  of  program  risk  assessment.  Since 
RARs  may  be  large  documents,  they  may  be  stored  as  files.  RARs  should  include  infor¬ 
mation  that  links  it  to  the  appropriate  RIF. 

2.3  RISK-HANDLING  DOCUMENTATION  (NOT  MANDATORY) 

Risk-handling  documentation  may  be  used  to  provide  the  PM  with  information  he  needs 
to  choose  the  preferred  mitigation  option  and  is  the  basis  for  the  handling  plan  summary 
contained  in  the  RIF.  This  document  describes  the  examination  process  for  risk-handling 
options  and  gives  the  basis  for  the  selection  of  the  recommended  choice.  After  the  PM 
chooses  an  option,  the  rationale  for  that  choice  may  be  included.  There  should  be  a  time- 
phased  plan  for  each  risk-mitigation  task.  Risk-handling  plans  are  based  on  results  of  the 
risk  assessment.  This  document  should  include  information  that  links  it  to  the  appropri¬ 
ate  RIF. 

2.4  RISK  MONITORING  DOCUMENTATION  (MANDATORY) 

The  PM  needs  a  summary  document  that  tracks  the  status  of  high  and  moderate  risks. 
The  XYZ  program  will  use  a  risk-tracking  list  that  contains  information  that  has  been 
entered  from  the  RIF.  An  example  of  the  tracking  list  is  shown  on  page  C-27. 
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3.0  DATABASE  MANAGEMENT  SYSTEM  (DBMS) 

The  XYZ  Risk  Management  Information  System  (RMIS)  provides  the  means  to  enter  and 
access  data,  control  access,  and  create  reports. 

Key  to  the  MIS  are  the  data  elements  that  reside  in  the  database.  Listed  below  are  the 
types  of  risk  information  that  will  be  included  in  the  database.  "Element"  is  the  title  of 
the  database  field;  "Description"  is  a  summary  of  the  field  contents.  The  Risk  Manage¬ 
ment  Coordinator  will  create  the  standard  reports  such  as,  the  RIF,  Risk  Monitoring,  etc. 
The  RMIS  also  has  the  ability  to  create  "ad  hoc"  reports,  which  can  be  designed  by  users 
and  the  Risk  Management  Coordinator. 

DBMS  Elements 

Element  _  Description _ _ 

Risk  l(jentification  (ID)  Identifies  the  risk  and  is  a  critical  element  of  information,  assuming  that  a 
Number  relational  database  will  be  used  by  the  PMO.  (Construct  the  ID  number  to 

identify  the  organization  responsible  for  oversight.) _ 

Risk  Event  States  the  risk  event  and  identifies  it  with  a  descriptive  name.  The  statement 

and  risk  identification  number  will  always  be  associated  in  any  report. _ 

Priority  Reflects  the  importance  of  this  risk  priority  assigned  by  the  PMO  compared  to 

all  other  risks,  e.g.,  a  one  indicates  the  highest  priority.  _ 

Date  Submitted  Gives  the  date  that  the  RIF  was  submitted. _ 

Major  System/  Identifies  the  major  system/component  based  on  the  Work  Breakdown 

Component  Structure  (WBS). _ 

Subsystem/  Identifies  the  pertinent  subsystem  or  component  based  on  the  WBS. 

Functional  Area _ _ _ 

Category  Identifies  the  risk  as  technical/performance,  cost,  or  schedule  or  combination  of 

these. _ _ _ _ 

Statement  of  Risk  Gives  a  concise  statement  (one  or  two  sentences)  of  the  risk. _ 

Description  of  Risk  Briefly  describes  the  risk;  lists  the  key  processes  that  are  involved  in  the  design, 

development,  and  production  of  the  particular  system  or  subsystem.  If 
technical/performance,  include  how  it  is  manifested  (e.g.,  design  and 
engineering,  manufacturing,  etc.). _ _ 

Key  parameters  Identifies  the  key  parameter,  minimum  acceptable  value,  and  goal  value,  if 

appropriate.  Identifies  associated  subsystem  values  required  to  meet  the 
minimum  acceptable  value  and  describes  the  principal  events  planned  to 
dem onstrate  that  the  minimum  value  has  been  met. _ 

Assessment  States  If  an  assessment  has  been  done.  Cites  the  Risk  Assessment  Report,  If 

appropriate. _ 

Analysis  Briefly  describes  the  analysis  done  to  assess  the  risk  and  includes  rationale  and 

basis  for  results. _ 

Process  Variance  States  the  variance  of  critical  technical  processes  from  known  standards  or  best 
practices,  based  on  definitions  in  the  program’s  risk  management  plan. 

Probability  of  States  the  likelihood  of  the  event  occurring,  based  on  definitions  in  the 

Occurrence  program’s  risk-management  plan.  _ _ _ 

Consequence  States  the  consequence  of  the  event,  if  it  occurs,  based  on  definitions  in  the 

program’s  risk-management  plan. _ 

Time  Sensitivity  Estimates  the  relative  urgency  for  implement  the  risk-handling  option. _ 

Other  Affected  Areas  If  appropriate,  identifies  any  other  subsystem  or  process  that  this  risk  affects. 

Risk-Handling  Plans  Briefly  describes  plans  to  mitigate  the  risk.  Refers  to  any  detailed  plans  that 
may  exist,  if  appropriate. _ 

Risk-Monitoring  Activity  Measurement  and  metrics  for  tracking  progress  in  Implementing  risk-handling 

plans  and  achieving  planned  results  for  risk  reduction.  _ 

Status  Briefly  reports  the  status  of  the  risk-handling  activities  and  outcomes  relevant  to 

any  risk-handling  milestones. _ 

Status  Date _ Lists  date  of  the  status  report. _ 

Assignment  Lists  individual  assigned  responsibility  for  mitigation  activities. _ 

Reported  By  Records  name  and  phone  number  of  Individual  who  reported  the  risk. _ 
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Risk  Inf  onnation  Form 


Risk  Identification  Number 
Risk  Event: 

Priority 

Major  System/Component/Functional  Area: 

Category: 

Statement  of  Risk: 

Description  of  Risk: 


Key  Parameters: 
Assessment: 

Analysis: 


Process  Variance 
Probability  of  Occurrence: 
Consequence: 


lime  Sensitivity: 

Other  Affected  Areas: 

Risk-Handling  Plans: 

Risk-Monitoring  Activity: 
Status 

Status  Date: 
Assignment: 


Date 


Reported  By: 
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RISK  TRACKING  REPORT 
(EXAMPLE  REPORT) 


Risk  Area  Status:  Design 

Pp  Hi 

Cp:  Hi 

Significant  Design  Risks: 

1.  Title:  System  Weight 

Pp  Hi 

Cp:  Hi 

Problem:  Exceed  system  weight  by  10%;  decreasing  the  range  and  increasing  fuel 
consumption. 

Action:  Examining  subsystems  to  determine  areas  where  weight  may  be  reduced. 
Reviewing  the  requirement.  Closely  watching  the  effect  on  reliability  and  survivabil- 
ity. 

2.  Title:  Design  Analysis  P^:  HiC^:  Hi 

Problem:  Failure  Modes,  Effects  and  Criticality  Analysis  (FMECA)  is  planned  too  late 
to  identify  and  correct  any  critical  single-point  failure  points  prior  to  design  freeze. 

Action:  Additional  resources  are  being  sought  to  expedite  performance  of  FMECA. 


Risk  Area  Status:  Supportability  Pp  Hi  Cp:  Mod/Hi 

1.  Title:  Operational  Support  Pp  Hi  Cp:  Mod/Hi 

Problem:  Power  supply  subcontractor  is  in  financial  trouble  and  may  go  out  of  busi¬ 
ness.  No  other  known  sources  exist. 

Action:  Doing  trade  study  to  see  if  alternative  designs  have  a  broader  power  supply 
vendor  base.  Prime  contractor  is  negotiating  with  the  subcontractor  to  buy  drawings 
for  development  of  second  source. 
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Watchlist  Example 


Potential 

Risk  Reduction  Actions 

Action 

Due  Date 

Date 

Explanation 

Risk  Event 

Code 

Completed 

•  Accurately 

•  Use  multiple  finite  element 

SE03 

31  Aug  97 

predicting  shock 

codes  &  simplified  numerical 

environment 

models  for  early 

equipment  will 

assessments. 

experience. 

•  Shock  test  simple  isolated 

SE03 

31  Aug  98 

structure,  simple  isolated 
deck,  and  proposed  isolated 
structure  to  improve 
confidence  in  predictions. 

•  Evaluating 

•  Concentrate  on  acoustic 

SE031 

31  Aug  97 

acoustic  impact 

modeling  and  scale  testing 

of  ship  systems 

of  technologies  not 

that  are  not 

demonstrated  successfully 

similar  to 

in  large  scale  tests  or  full 

previous 

scale  trials. 

designs. 

•  Factor  acoustic  signature 

SE032 

31  Aug  98 

mitigation  from  isolated 
modular  decks  into  system 
requirements.  Continue 
model  tests  to  validate 

predictions  for  isolated 
decks. 
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APPENDIX  C 

GLOSSARY 


AMSAA 

Army  Materiel  System  Analysis  Activity 

APB 

Acquisition  Program  Baseline 

ASP 

Acquisition  System  Protection 

BCS 

Baseline  Comparison  System 

CAIG 

Cost  Analysis  Improvement  Group 

CAIV 

Cost  As  an  Independent  Variable 

CCDR 

Contractor  Cost  Data  Reporting 

CPM 

Critical  Path  Method 

CWBS 

Contract  WBS 

DAD 

Defense  Acquisition  Deskbook 

DAU 

Defense  Acquisition  University 

DBMS 

Database  Management  System 

DCMC 

Defense  Contract  Management  Command 

DoD 

Department  of  Defense 

DoDD 

DoD  Directives 

DTSE&E 

Director,  Test,  Systems  Engineering,  and  Evaluation 

DT&E 

Developmental  Test  and  Evaluation 

EAC 

Estimate  At  Completion 

EMD 

Engineering  and  Manufacturing  Development 

ESC 

Electronic  Systems  Center 

EV 

Earned  Value 

GAO 

Government  Accounting  Office 

IBR 

Integrated  Baseline  Review 

IPPD 

Integrated  Product  and  Process  Development 
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IIPT 

Integrating  IPT 

IPTs 

Integrated  Product  Teams 

KPPs 

Key  Performance  Parameters 

LCC 

Life-Cycle  Cost 

LFT&E 

Live-Fire  Test  and  Evaluation 

MAIS 

Major  Automated  Information  System 

MDA 

Milestone  Decision  Authority 

MDAPs 

Major  Defense  Acquisition  Programs 

MIS 

Management  Information  System 

MNS 

Mission  Need  Statement 

MS 

Milestone 

OIPT 

Overarching  IPT 

ORD 

Operational  Requirement  Document 

OT&E 

Operational  Test  and  Evaluation 

POM 

Program  Objective  Memorandum 

PM 

Program  Manager 

PMO 

Program  Management  Office 

PMWS 

Program  Manager's  Work  Station 

PRAG 

Performance  Risk  Assessment  Group 

RAR 

Risk  Assessment  Report 

RFP 

Request  for  Proposal 

RIF 

Risk  Information  Form 

SEI 

Software  Engineering  Institute 

SOW 

Statement  of  Work 

SPMN 

Software  Program  Managers  Network 

SRE 

Software  Risk  Evaluation 

STAR 

Special  Threat  Assessment  Report 
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T&E 

Test  and  Evaluation 

TAAF 

Test-Analyze-and-Fix 

TEMP 

Test  and  Evaluation  Master  Plan 

TPM 

Technical  Performance  Measurement 

USD(A&T) 

Under  Secretary  of  Defense,  Acquisition  and  Technology 

WBS 

Work  Breakdown  Structure 
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